Category: BSD Operating Systems

Zombies, Sparks, and Computer Failure Diagnosis – a North-of-Hollywood Tale

For all of the manual pages, handbooks, HOWTOs, RFCs, tutorials, and op-ed manuscripts that I’ve read, about UNIX operating systems and “Known good” applications of UNIX operating systems – as across a short and canonically informal study of such technical concepts – I don’t recall having ever seen a single article about system diagnostics. The topic occurred to my own simple consideration, one evening in a chat channel, when one of the correspondents in the chat channel was commenting about a difficulty he was encountering in an application of a FreeBSD operating system, a difficulty about which he was seeking some of the good-old free and open source technical support.

Considering the comprehensive nature of the FreeBSD Handbook – such that, in any depth of study, the Handbook may be accompanied with a study of the manual pages of shell command line applications denoted in the Handbook – moreover, considering albeit in a broad way the abstract library of books written about FreeBSD, and the further depths of detailed knowledge provided of such veritable tomes of non-arbitrary writing, I don’ t believe there is any sort of a vacuum in documentation about FreeBSD. Perhaps not a lot of the documentation would appeal to any great sense of immediate convenience, however.

If it may perhaps seem like a niche concern, socially, to develop any manner of formal documentation about diagnosing an application of a FreeBSD operating system, of course there may be any times when – by whatever convergence of cosmic rays, etc. – times when an application of a FreeBSD operating system – broadly, when any computer operating system may not be performing in all ways as may be hoped or assumed, and the divergence between the “Observed results” and the “Ideal results” of a magnitude that may serve to require some consideration.

More simply so, there may be times when any single application is “Not Working Out,” such that the state of it “Not Working Out” would be realized with any number of additional events, such as:

  • Error messages that may have been printed to a console stream – whether printed to the standard output stream or standard error stream, as both being typically available in UNIX applications
  • Log data that may have been printed to any one of the System Log files
  • Sparks that may have been flying out of any unexpected locations in the computer enclosure
  • Zombies that may have been crawling out from under the floorboards
  • Movie directors that may have been writing the Next Big Blockbuster, about it all

Whether or not it may seem immediately clear, as for how any such  events may be occurring in a known cosmos, and in such that the issue could then be further diagnosed to its causes, but at least the error messages and the log data may be reproducible, in many instances. Sometimes, of course, there may be the even more exceptional instances – such as, in which the zombies might be crawling in from the rafters, the Movie directors instead becoming situated under the floorboards, and neither a console error message or a log message being emitted about whatever the computer is doing, at the time.

So, on a good day – or, at least, a moderately good day, but in such as would be interrupted by any number of unexpected failures in a software system, among all of a life’s events – in so much us a lukewarmly good day, so, it might seem as if all the rigorous diagnosis work was some kind of a mundane task? When the Zombies and the Directors arrive, so to speak, “All bets are off.”

Until such time – and in so far as that a computer implements a manner of a finite state machine such that may be logically observed for all of its systematic behaviors – then when any of those same behaviors would comprise, logically, systematic failures, likewise the finite system of it can be diagnosed logically as to why the systematic failures of it would be occurring – all the more easily so, when any error messages and any log data produced of the failure or failures would be available for the diagnosis.

Sure, in the long and the short of it, it might not make a topic for any exactly patented kind of an approach to Solutions Building. If it may not make a topic for any great sort of a voluminous reference manuscript, furthermore, but the work of diagnosing an unexpectedly failing computer system – in all of the innate, perhaps necessary simplicity of it – a diagnosis may be developed, itself, as a diagnosis about such a manner of a state of a computer system. That is to say, when a computer system is in any manner of a very broad “Does Not Work” state  there may be some very simple procedures, such as denoted in the previous.

Leaving out the Zombies, the sparks, and the Movie directors momentarily:

  • Console Messages
  • Log Data

Those very simple things can be of an immeasurable value when diagnosing a computer software failure.

Towards an Overview about IPFW on FreeBSD, in an Application Entailing a Diagnosis about a “Deep Configuration Issue”

As principally a concept of philosophical schools, logic may serve in a role in numerous material vocations – such as in any number of legal domains, material vocational domains, and financial domains of vocational practices. Ostensibly, logic may also find a role in any number of more nuanced vocations, such as in prcactices of social sciences and in creative fields of the arts. It may seeem a trivial thing to denote, thus, that a sense of logic may be derived of the technically practical syntax and procedural semantics of any single network firewall configuration framework.

Presently, this article will diverge from the broadly philosophical view as to focus principally about the IPFW firewall architecture, as IPFW providing one available packet filtering interface on a typically built FreeBSD operating system. As the FreeBSD kernel provides a complete firewall architecture corresponding with OS kernel services that may be configured with the ‘ipfw’ shell command and other, corresponding shell command line utilities, this article will endeavor to provide albeit a broad topical overview about services architectures as corresponding immediately with the IPFW shell command line interface – herein, denoted as the IPFW CLI. In this article, albeit broadly, the IPFW architecture will be juxtaposed to the PF and IPF firewall frameworks, such that are also available – as in a manner of a parallel corespondenance to IPFW – in the canonical FreeBSD kernel.

In short synopsis, FreeBSD provides three distinct firewall architectures – such that a network systems administrator may select from, in any development of any number of individual network systems applications. Although it may seem, in a manner, modestly mystifying to a novice administrator, but in respects to the state of the art in FreeBSD kernel applications – such a diversity of firewall frameworks, in a single operating system – it may certainly permit for any manner of a scholastic survey of the individual firewall frameworks, thus available. All three of the individual firewall frameworks – as denoted – are documented in the FreeBSD handbook, such that may serve as a fortuitious manner of a “Starting Point,” for any scholastic review as such.

Administrators’ opinions may vary, as to the nuances of individual firewall designs and of corresponding firewall applications. In the opinions of the author of this article, PF – for instance – provides for a novel manner of a firewall scaling function, such that may be of some use for any number oc applications in rural wireless broadband internetworking – such as, in digital networks applying any manner of a commercial satellite internetworking service – if not also of use in some mobile-to-mobile (M2M) and Internet of Things (IoT) wireless networks. Specifically, the ‘high-latency’ option for PF firewall configuration provides a feature of a seemingly unique nature, across the proverbial “Gang of Three” of firewall architectures available on the FreeBSD OS. Certainly, this may not be the only unique feature that may be discovered of any scholastic survey of the FreeBSD firewall space. By only a small coincidence of geography, it occurs to the considerations of the author of this article, as so.

If in any instance of migrating a firewall configuration across any two of the three available firewall frameworks available in FreeBSD, perhaps any manner of a logical model of firewall services may be developed, as in a manner apropos to the task. As a a network firewall representing a manner of a feature of an operating system – broadly, as a feature of open source softwade systems – a firewall may be studied as a manner of a development onto a framework of packet-oriented communications networking. Thus, one may begin at the conventional OSI Seven Layer Model, such that may be studied as towards developing a manner of a principally vendor-neutral logical model. The OSI Seven Layer Model – broadly – as having developed as a feature of the “State of the Art,” in packet-switched networking services, the same model provides a manner of a normative domain-oriented view about network architectures.

Towards developing a depth of a detailed view, as to how the OSI Seven Layer Model is effectively reified in individual network systems architectures, of course one may begin at the simple characteristics of network media – whether of physical network media or wireless network media, and whether of electromagnetic wireless media or of any novel, if not experimental instances of optical wireless media. As such, the logical model could entail a definition of any number of commonly known digital network signaling protocols, if not towards a principally channel-oriented definition of network media. Thence, any number of software services may be defined to the model, such as begininng at the “Next lowest level” – such as viz a viz logical data types for network interface addressing and procedures of network address resolution in individual network topologies, viz token ring, star, or other topological models. Thence, the interface addresses may be represented in the model as for how those are effecitvely extended with any higher-level addressing schemes, at which point there are the “Lowest level” of network packet formats. Proceeding as to model a TCP/IP network, the model may then be developed as with regards to name resolution services, and thence, secure tunneling and data compression services. Lastly, the model may be extended as to represent each of the connectionless datagram models of UDP applications and the stateful message-oriented models of TCP applications, for how each may be finally realized in a network application protocol such as HTTP or IIOP.

Thus, perhaps it may be understood that any single concrete reification of the OSI Seven Layer Model – essentially, a topical model – may be non-trivial to develop as to any appreciable depth of concrete logical representation. Albeit as a manner of a principally orthogonal diversion from the parimary topical focus of this article, perhaps the outline presented in the previous paragraph may serve towards at least a broad overview of how the OSI Seven Layer Model is realized in any number of practical network systems applications.

As a network firewall effectively residing – in an abstract manner – residing in an application of the OSI Seven Layer Model, but the firewall itself must apply any number of concrete network services, such that would be available as services of any single operating system (OS) in which the respective firewall would be implemented. Thus, beside all of the semantic nature of the OSI Seven Layer Model, but a firewall is implemented as principally a software service in any single operating system. Returning attention, thus, from addressing the vendor-netural logical model – momentarily, avoiding any far detailed description of how the OSI Seven Layer model is effectively realized in FreeBSD – this article returns attention to a domain of computer operating systems, after a brief aside with regards to applications of operating systems. Certainly, no single OS model resides as though in a vacuum.

Any single vendor-neutral architectural model might not as well survive a transition to commercial marketing practices, once addressed into a domain of commercial software development. As with regards to a logical model of TCP/IP networking, perhaps some commercial software developments may serve to not obscure the nature of the underlying architecture – contrasted to any more intensively brand-oriented models, as in which any single concept of software as product may effectively serve to obscure any single of concept of software as a concretely logical construct. However, not every single commercial software product campaign may serve to inherently confuse the nature of software systems design.

Thus, there are the Microsoft products, the OS X products, and the products of so many lesser known commercially branded operating system sales programs, and then there are the architectures of the software products that those same sales programs may endeavor to derive any manner of a commercial capital about. There are also so many free/open source operating systems, ostensibly developed as independent of single capital sales and marketing programs.

In a manner semantically narrower than of any individual “Products Domain,” thus – more narrowly – any single manner of an “Applications Domain” may be observed, such as in a scholastic study of applications of computing services in human society. This article will not propose any single logical model of the latter – that the details of any single concrete, technical architecture may be tedious enough to describe, and that the logical if not creative semantics of any single socio-ethnographic model may seem no less demanding for its scholastic nature. Certainly, not all things of human society may devolve to a statistical oversimplification.

That it is in human society – broadly, thus – where computing systems are applied, of course so many applications of computing may not seem to devolve entirely to a statistical model of material capital. The author of this article proposes that the human nature of applications of computing may seem to be most apparent in a context of any single Small Office/Home Office (SOHO) computing environment – and that, as such, perhaps a SOHO LAN may not seem in all ways analogous to any corporate commercial server farm.

Thus, this article will be developed not as though to focus about the high-volume and no doubt very capital-intensive environment of any manner of a prototypical commercial network. Neither will this article endeavor to present any manner of a tourist’s view of commercial academia – the strange hybrid of marketing and scholarly discourse that develops as of the heavily marketed technologies of individual products developed in a context of computing.  Certainly, there was an era when computer systems were developed for not so much of a popular style, as much as for a scientifically reproducible state of the art – the veritable “Old Scrolls” of computer science might serve to provide something of a sense of context to that era. Perhaps it is an era not yet all or entirely lost to contemporary practitioners of product engineering and commercial systems development.

By now, this article has diverged to something of a semantic distance aside to the original topic, the IPFW architecture as a feature developed of the FreeBSD operating system. Not as though to abandon the original thesis concept, simply it may seem as though to trivialize the State of the Art, if any technical material about computing would be presented without a substantial sense of context, as towards a regard about the origins of the products therein applied.

So as to present the following overview beside a further sense of a contextual note as with regards to computer systems diagnostics: Though the marketing and the documentation developed about so many commercial products products may seem, in itself, somehow mystifying,  but if there may be any singular concrete sense of nature developed as about computer science, it may theoretically be possible to develop a logical model of any single computer software and hardware product – no matter the aim of the modeling task, and no matter for the depth of marketing about the respective product.  Thus, Microsoft Active Directory can be presented in albeit a “Buzkill” manner of view as it representing an amalgam of LDAP, DNS, optionally SQL, optionally Kerberos, and other existing network application protocols all completely described in open, accessible standards – up to the point in which the same protocols are implemented in the singular product, as such.

In a sense, a simple task of defining a logical model of a formal software system may serve – in a regards – towards developing a logical diagnosis of any feature of “Unexpected functionality” or lack of “Expected functionally” in the system thusly modeled. In that the modeling task, as such, may endeavor to develop a manner of a platform-agnostic view of the system being  modeled, perhaps it may at least serve to limit any bias of opinion about the model and its reification.

Firewall configuration – as a feature consolidated directly of the design of an operating system kernel – may not be subjected to as much standardization as so many platform-agnostic communication protocols, application data channel interfaces, and hardware forms yet developed to the modern domain of computing. Thus, this article may not suppose to contribute any manner of a structural perspective for international standards organizations. The simple concept of a “Vendor standard” manner of firewall implementation may have to suffice.

Not, then, as though to oversimplify the lateral development of the IPFW firewall architecture in FreeBSD, but to begin at a more practical note: The IPFW command line interface – though it is not all there that there is to the IPFW architecture – rather, the documentation about the IPFW command line interface – as towards a sense not only for firewall configuration definition, but also for diagnostics about firewall configurations – would be lost to this article, perhaps, if this article had begun as so:

In a personal review of my LAN’s present IPFW configuration, I can’t seem to figure out what in my LAN gateway’s IPFW configuration is resulting in some packets getting “lost” – as in any HTTP requests made on the LAN and without a proxy. The concrete nature of that issue, as I am presently seeing it, is that the packets are not being blocked by the firewall – that there is no any kind of a “Connection refused” message being produced to the unproxied HTTP requests, they’re simply resulting in timeout and varaibly “No route to host” (via ‘fetch’) or “Operation time out” (via curl) error messages. Therefore, I believe that the packets are being lost, and I believe that it may be a consequence of the firewall configuration.

The issue is not a very high-priority issue – I can still use HTTP services on the LAN, so long as those HTTP services are communicated through the HTTP proxy on the LAN. For some applications, however, that may not be sufficient. In a manner of a personal sense, I do not believe it is sufficient that I am unable to understand “Why?” as to the nature of the packet lost in unproxied HTTP requests, juxtaposed to the present configuration of the IPFW service on the LAN gateway appliance.

To diagnose this issue, I am not foremost interested in taking a popular scalpel to my LAN’s present IPFW configuration. Thus, I will not be addressing the issue to the forums. Personally, I am not even greatly worried about the issue – it does not serve to interfere with the LAN’s primary functional applications – but is a logical conundrum, moreover a manner of a practical “Show stopper” to some applications on the network.

The IPFW CLI, in itself, provides a manner of a shell-oriented application programming interface for firewall configuration management. On review of the manual page for IPFW, it becomes apparent that there are more topical features of IPFW than have been presented immediately in the practical introduction in the Handbook. This, then, is not a criticism of the FreeBSD handbook – rather, this is a simple effort in extending beyond the handbook’s own imminently practical domain of discourse, while endeavoring to diagnose, sole, and describe an albeit non-blocking issue presently being encountered with regards to my own LAN’s gateway firewall configuration.

Of course, this oblique style of writing would not be sufficient for addressing the matter to a task. If there may be a long-and-short introduction to a task of network systems diagnostics, perhaps this article may at least serve as a manner of a rhetorical overview about the matter. A description of the technical details of it may be begun in a separate article, shortly.

Tips for SSH in FreeBSD Userspace – Keyfiles from PuTTY; ssh-agent in login shell

OpenSSH Keys from PuTTY

Extending on a model presented at StackExchange *, after rebooting to the FreeBSD desktop OS on my notebook PC, I’ve developed a short shell script as for a purpose of reformatting the private component of my SSH key files as were generated with PuTTY, such that the output of the reformatting tool may be applied with OpenSSH and ssh-agent on FreeBSD.

#!/bin/sh

for F in *.ppk; 
  do NM=${F%.ppk};
  ## print file name before puttygen queries for password
  echo Converting key $NM 
  puttygen $F -O private-openssh -o $NMafter
done

This shell script is designed for an application in an SH-compatible shell. The shell script applies one of the extended parameter substitution features – as available with SH and similarly, available with BASH – such that the ${F%.ppk} expression is evaluated by the shell script interpreter, producing a string value that is the name of the file referenced in $F, sans the file name’s type string.

The shell script – as a quick “One shot” SH snippet – is written as to be run by a user, from within the user’s ~/.ssh directory. The shell script processes all PuTTY *.ppk files in the current directory, as to generate a key file in a format for OpenSSH, for each file – overwriting any existing files, in the process.

SSH in Login Shells

The following shell script may be useful for running ssh-agent, as from within ~/.shrc – in reference to SH ENV – or in ~/.bashrc, there as in reference to BASH_ENV.

if [ -z "$SSH_AGENT_PID" ] || 
  ! [ $(ps -p $SSH_AGENT_PID -o comm) = ssh-agent ]; 
# start SSH agent if not already running 
  then eval $(ssh-agent) 
fi

The previous shell script will initialize an ssh-agent process in the immediate shell environment, if the shell script computes than an ssh-agent is not presently running.

Corresponding to the pregvious  shell script snippet, the following may be recommended as a feature for ~/.bash_logout

if [ -n "$SSH_AGENT_PID" ] && 
    [ $(ps -p $SSH_AGENT_PID -o comm) = ssh-agent ];
then eval $(ssh-agent -k)
fi

The previous shell script – if included  in ~/.bash_logout – will terminate the ssh-agent process in the immediate shell environment, on logout from a BASH login shell, if the shell script computes than an ssh-agent was running in the immediate shell environment of the login shell. ** Such a feature may be applied as to ensure that the ssh-agent  process will be terminated ,  subsequent to when the user has logged out of the host. Otherwise, ssh-agent might continue to run as a forked process, independent of any controlling terminal.

* More info: How to convert SSH keypairs generated using PuttyGen(Windows) into key-pairs used by ssh-agent and KeyChain(Linux)

** More info:  bash(1) – bash – GNU Bourne-Again SHell, section: Invocation

A Rambling Introduction to a Linear Experience of Computing

Not as though to begin any more of the Web’s Favorite or Least Favorite brands of Identity Politics, Online, but rather as to entertain a simple rhetorical question: How to even begin to describe any of the adventures I’ve ever known and known personally about  UNIX computing? Where could I even start, about such a topic? If it may be in any ways a personal story, but as I am not so much a fan of vicarious descriptions of personal experience, I may not believe there is so much to say of it.

In a technical sense – in a very short synopsis – in regards to Linux operating systems, singularly, I’ve applied each of Slackware, Debian, SuSeDebian again, Ubuntu, and Debian again, for any appreciable duration at a desktop computer – OpenShift, for a short while, it is a kind of a  cloud-oriented Linux computing environment. There’s also Android, of course – an operating system developed on a Linux kernel – moreover a project as primarily sponsored by Google, secondly sponsored by individual Original Equipment Manufacturer (OEM) designers, beside  the Android Open Source Project. Then there’s Tizen, also –  something of a newer Linux operating system, as primarily sponsored by Samsung – and  the Maemo operating system, as was sponsored by Nokia.

Sometime in the duration of applying Linux on a desktop, I’ve also taken at least a topical look at QNX Momentics and the QNX Photon GUI, as was at a time when those were being made readily available for limited licensing on desktop personal computers. I’d say that I’m still impressed with the QNX Real Time Operating System (RTOS), but of course it’s not a free/open source software (FOSS) operating system. Perhaps it may not seem to permit for much of a sense of discussion, therefore, as in regards to all of the intellectual property presently invested in QNX.

Personally, I’ve yet to make any application of Solaris, but of course there is the Illumos OS family tree, however – to which, as perhaps a manner of a complimentary concept, there’s even an OpenSparc System on a Chip (SoC) project, developed in a manner of open access microcontroller design.

Not as though to indulge too far about any high-level concepts or low-level applications in regards to any singular contexts of computing,  but it may seem to be simply an ideal time in the history of the world as a duration in which one may endeavor to be a student of information sciences and computing.

This year,  personally – in by in large departing from a momentary study of the Cisco IOS operating system and its newest QNX baseline – I’ve begun to study the FreeBSD operating system. This week, moreover, I’ve begun to make an attention about the NetBSD pkgsrc framework – specifically, as with an interest in adapting pkgsrc for userspace software distribution on FreeBSD. This, of course, may not turn out to be too much of a trivial task – not as trivial of a task as its logical model may seem to suggest, in any single and immediate estimate – but it may well be “Worth it.”

So many applications of free/open source operating systems, and so many approaches in developing a concept of computing. It’s certainly a long ways away from tooling around with Tandy Deskmate, wouldn’t it seem?

 

Notes – Specialized FreeBSD Workflows – FreeBSD for AWS EC2, FreeBSD for Embedded Computing

Concerning applications of the FreeBSD operating system on the Amazon Web Services (AWS) cloud services architecture – specifically in the AWS Elastic Compute Cloud (EC2) – and applications of FreeBSD on embedded computing platforms, a few very short notes:

Conceptual Overview

  • Concept: Workflow
    • Workflow e.g. for building a FreeBSD distribution for local LAN
      • Some characteristics of Note
        • Objects of work: “Bare metal” computing hardware (PCs, Racks); “Twisted pair”, “Fiber”, and “Wireless” network media
        • Hardware accessible to administrative team
        • Installation may be managed in correlation with networked file services – including NFS – and networked credential services, viz a viz YP/NIS
    • Workflow e.g. for building a FreeBSD distribution for installation to AWS EC2
      • Characteristics of Note
        • Objects of work: EC2 virtual machine instances; S3 and EBS storage modules; RDS data services; other modules in AWS modular service mix
        • Hardware not accessible – juxtapose to virtual console as available with Digital Ocean
        • AWS Virtual Private Cloud (VPC) networking – IP address allocation, address assignment, subnetting, routing all configured via VPC API – largely independent of indiviual EC2 virual machine instances
        • Credential services via AWS IAM
    • Workflow e.g. for building a FreeBSD distribution for installation to an embedded computing platform
      • Characteristics of Note
        • “Bare metal” computing hardware (single-board computers, systems on chip, systems on module)
        • Hardware components – once configured – may be applied without immediate network availability
        • Functional systems tests may be exceptionally important – e.g. functional unit testing for new software components; integration testing for all software components, in emulator
  • Concept: Modularity
    • Not just a buzzword: Component-Based Software Engineering (CBSE)
    • Platform modeling – cf. Cadena – may be assistive in phases of design, testing, and in application phases
  • Concept: Documentation
    • Topically focused documentation schema: DITA

Technical Resources

  • EC2 AMI: FreeBSD 10
    • Ed. Notes
      • UFS Root Filesystem
      • SCHED_ULE
        • EC2 t2.micro instances provide only one virtual processor core to each instance. Therefore, the 4.4 BSD scheduler may be appropriate for such instances
      • sysrc parameters available via configinit
      • IP addressing via AWS VPC
  • Configuration File: /usr/src/release/release.conf.sample
  • Makefile/usr/src/release/Makefile.vm
    • Application: Makefile is sourced in correlation with release.conf
  • Makefile: /usr/src/release/Makefile.ec2