HellHound Documentation

Contemporary software engineering still has a lot to learn from the 1970s. As we’re in such a fast-moving field, we often have a tendency of dismissing older ideas as irrelevant—and consequently, we end up having to learn the same lessons over and over again, the hard way. Although computers have become faster, data has grown bigger, and requirements have become more complex, many old ideas are actually still highly relevant today.

In this section, I’d like to highlight one particular set of old ideas that I think deserves more attention today, the Unix philosophy.

Systems composed by [Components]. A system knows how to start and stop components. It is also responsible for managing dependencies between theme. Components are the smallest parts of a system whic are reusable.

A HellHound system is basically a map describing different aspects of a program. Each program might have several systems. For example a development system and a production system. But there would be just a system running at a given time.

Systems should follow a certain spec (:hellhound.system/system). For example any system should have a`:components` key with a vector as its value. All the componentsof the system should be defined under this key. Each component is a map describing the behaviours of that component (For more info on components read [here](./Components.md)).

In order to use HellHound systems. First you need to define your system by defining a map with at least one key. Yupe, you guessed it correctly, the :components key. We’re going to discuss the system’s keys in bit. So for now just let’s talk about how to start and stop a system.

After defining the system. The next step would be to set the defined system as the default system of your program by using hellhound.system/set-system! function. This function accepts just one argument which is the system map. It basically analayze the given map and set it as the default system of the program. From now on, you can call hellhound.system/start and hellhound.system/stop function.

Basic execution flows of a system are start and stop. Both of them creates a graph from the system components and their dependencies. Then they start/stop the system by walking through that graph and calling the start-fn and stop-fn function of each component.

After starting a system, you can access the running system via hellhound.system/system function. It would returns a map describing the current running system including all the running components.

There are two ways to manage dependencies in your components code. The first one via a the system map itself, by treating the system map as a repository of dependencies. The second way is via dependency injection model which is not supported at this version (v1.0.0). Anyway, any component can pull what ever it needs from the system map by calling hellhound.system/get-component function and passing the component name to it. This function will simply look for the component in system map and returns the running component map. There is a huge problem with this apparoach. By using this apparoach components knows too much about the environment around them, so it reduce their portablity.

System’s workflow describes the dataflow of the system. It describes where data comes in and where information goes out of the system. By "where" I mean which component of the system. As mentioned before each HellHound [Component](./Components.md) has an input stream and an output stream assigned to them. You can think of each component as a pipe, the whole system as a pipeline. Data flows to this pipeline based on your piping (description) from entry points and exits the pipeline from exit points. Pipeline might have more than one entry or exit points or none at all.

Using the system workflow you can design an open or a close dataflow for your system. A Close system is a type of system which all of its components have their input and output connected. In the other hand, an open system is a type of system which NOT all of the inputs and outputs of components connected to each other.

Components of a system should consum from their INPUT and produce their OUTPUT in non-blocking fashion in order to avoid blocking in a system.

Check out the [workflow](./Workflow.md) section for more information.

Large applications often consist of many stateful processes which must be started and stopped in a particular order. The component model makes those relationships explicit and declarative, instead of implicit in imperative code.

Components provide some basic guidance for structuring a HellHound application, with boundaries between different parts of a system. Components offer some encapsulation, in the sense of grouping together related entities. Each component receives references only to the things it needs, avoiding the unnecessary shared state. Instead of reaching through multiple levels of nested maps, a component can have everything it needs at most one map lookup away.

Instead of having mutable state (atoms, refs, etc.) scattered throughout different namespaces, all the stateful parts of an application can be gathered together. In some cases, using components may eliminate the need for mutable references altogether, for example, to store the "current" connection to a resource such as a database. At the same time, having all state reachable via a single [system](./README.md#overview) map makes it easy to reach in and inspect any part of the application from the REPL.

The component dependency model makes it easy to swap in stub or mock implementations of a component for testing purposes, without relying on time-dependent constructs, such as with-redefs or binding, which are often subject to race conditions in multi-threaded code.

Having a coherent way to set up and tear down all the state associated with an application enables rapid development cycles without restarting the JVM. It can also make unit tests faster and more independent, since the cost of creating and starting a system is low enough that every test can create a new instance of the system.

For small applications, declaring the dependency relationships among components may actually be more work than manually starting all the components in the correct order or even not using component model at all. Everything comes at a price.

The [system](./README.md#overview) map produced by HellHound is a complex map and it is typically too large to inspect visually. But there are enough helper functions in hellhound.system` namespace to help you with it.

You must explicitly specify all the dependency relationships among components: the code cannot discover these relationships automatically.

Finally, HellHound system forbids cyclic dependencies among components. I believe that cyclic dependencies usually indicate architectural flaws and can be eliminated by restructuring the application. In the rare case where a cyclic dependency cannot be avoided, you can use mutable references to manage it, but this is outside the scope of components.

Components are the main parts of HellHound systems. Basically, each component is an implementation of IComponent protocol. The protocol which defines a component functionality. By default HellHound implements IComponent protocols for hashmaps only. So we can define components in form of maps.

In order to define a component, a map should contain the following keys (All the keys should be namespaced keyword under hellhound.component):

So as an example:

As you can see creating a component is fairly simple and nothing special.

As you already know, start-fn of a component takes two arguments. The first one it the component map itself and second one is a hashmap called context map.

The purpose of a context map is to provides co-effects for the component, Such as dependency components. For example HellHound inject a key called :dependencies to the context map which is a vector containing those components that this component is depends on. The order of this vector would be exactly the same as :depends-on value in component description map. So you can get all you need from the context map. You can think of it as some sort of dependency injection for you component.

But context map contains more data that you might need them in action. Here is the list of keys of a context map:

Sometimes, we need to dispatch values to a component conditionally. For example imagin a system that is responsible for separeting odd numbers from even numbers in a stream of numbers. Checkout the following workflow definition:

As you can see in the workflow definition above, it’s possible to describe a condition for dispatching values from a component to another one. In the previous example all the values from :component-1 flow to :component-2 only if the provided condition returns true for each incoming value which in this case only odd values are going to deliver to component-2. The same applies to the second pipe but only even values flow to component-3.

The predicate function can be any function which receives an argument ( The value from upstream component ) and returns a boolean value indicating whether the value should flow to the downstream component or not.

Base on what we discussed up until now we can test our workflow like this:

The output of the above snippet would be like:

<PLACEHOLDER TEXT> Explaination about predicate best practices and hellhound.messaging ns

First of all let’s create a very simple component with just one dependency to have a minimum working environment.

In order to execute the following example from the hellhound_examples repository. Just issue the following command in the repository root directory.

Ok let’s jump into the code.

When we execute the above code the output would be something like:

In this example we’re going to create a really simple system with a simple linear workflow. The workflow is really simple, data will flow from component-1 to component-2 and then component-3. Each component transoform the value and put it to the output.

In order to execute the following example from the hellhound_examples repository. Just issue the following command in the repository root directory.

Ok let’s jump into the code.

When we execute the above code the output would be something like:

As you can see the output above data flows to the pipeline and each component transoforms the value it gets from its input to a new value and put it into the output.

In the above example we used a simple vector and converted it to a stream source, but in practice the data flows to our system from an external source like a web server.

This example is dedicated to conditional workflow. A quick demonstration of how you can setup a conditional workflow in your system.

In order to execute the following example from the hellhound_examples repository. Just issue the following command in the repository root directory.

Ok let’s jump into the code.

When we execute the above code the output would be something like:

The above output is clear enough. Component 2 got only odd values and component 3 got the even values. The predicate functions of a workflow catalog can be any function. But you need to bear in mind that HellHound is going to run these predicates for each value so for the sake of performance we need to keep the fast and pure.

<PLACEHOLDER TEXT>

The HTTP package of HellHound is made from several smaller parts which work together. Parts such as HTTP router, Websocket server, Event Dispatcher and so on.

In general sense the HellHound’s HTTP stack is just a barbone Ring compatible Pedestal webserver. The recommended way of using it is through hellhound.components.webserver/factory function with no parameter which creates a webserver with all the default values which in most cases you don’t need to chaing. By default HellHound will spin up a webserver which only serves assets, root enpoint and a websocket endpoint and the rest of the process happens on the client side application (Which handles by HellHound again), So most of the time when you need a new route in your application, you have to add it on the client side router. Ofcourse you can choose to not follow the same pattern but then you won’t get enough benefit of using HellHound.

HTTP package of HellHound has been composed by several pieces. In its heart it uses the Pedestal’s Interceptors to reply to HTTP requests. So almost all of the Pedestal concepts applies to HellHound HTTP as well.

Another key part of HellHound HTTP package is the HTTP router. There are plenty or Clojure routers around but in my opinion the best of is the Pdestal Router. It’s a data driven HTTP router with is simple, and predictable. We’ll discuss Pedestal router later in this chapter.

The most important part of HTTP package is the webserver component which lives in hellhound.components.webserver. Under the hood it uses Aleph that is an asynchronous network library on top Netty.

<PLACEHOLDER TEXT>

HellHound distinguishes between routes and routers.

"Routes" are data that describe a structure of decisions to be made in handling requests.

"Routers" are the functions (supplied as Interceptors) that analyze incoming requests against the routes.

Pedestal uses protocols to connect routes to routers. Either can be varied independently.

Routes are data used by routers. The verbose syntax is the form used directly by routers. The table and terse syntaxes are convenient forms that can be expanded into the verbose syntax.

Users are free to describe routes however they like, so long as the data that reaches the router is in the verbose syntax.

The built in syntaxes are expanded by expand-routes polymorphically on the argument type:

The verbose syntax is a list of maps, with the following structure.

:route-name must be unique.

The keys :path-re, :path-parts, :path-params, and :path-constraints are derived from the :path.

Users will not generally write routes directly in verbose format.

Table syntax is expanded by expand-routes into the full (verbose) syntax shown above.

When the argument to expand-routes is a set, it will be expanded using the table syntax.

In table syntax, each route is a vector of:

The :host, :port, :app-name, and :scheme are provided in a map that applies to all routes in the list.

When multiple routes use the same path, they must differ by both verb and route name.

If the last interceptor in the chain has a name, or you supply a symbol that resolves to a function or named interceptor, then the route name will be derived from that.

Terse syntax is expanded by expand-routes into the full (verbose) syntax shown above.

When the argument to expand-routes is a vector, it will be expanded using the terse syntax.

In the terse format, a route table is a vector of nested vectors. Each top-level vector describes an "application". The application vector contain the following elements:

Route vectors can be nested arbitrarily deep. Each vector adds a path segment. The nesting structure of the route vectors maps to the hierarchic tree structure of the routes.

Each route vector contains the following:

The allowed keys in a verb map are:

The value of each key is either a handler function or a list of interceptors.

Each verb in the verb map defines a route on the path. This example defines four routes.

Interceptor metadata applies to every route in the verb map. In this example load-order-from-db applies to both the :get and :put routes for the path "/order/:id"

(Recall that metadata is attached to the next data structure read. The metadata with constraints and interceptors will be attached to the verb map.)

If multiple routes have the same handler, you will need to distinguish them by providing route names. (This is necessary so URL generation knows which route to use.) A route name is a keyword that goes in the first position of an interceptor vector in the verb map. In the following example, both POST routes have the same handler. We provide the keywords :post-without-id and :post-by-id to distinguish the routes.

When the value of :io.pedestal.http/router is a function, that function is used to construct a router. The function must take one argument: the collection of fully expanded routes. It must return something that satisfies the Router protocol.

In this example we’re going to create a really simple system which contains a hellhound webserver which reponds with what is gets via the ws connection.

In order to execute the following example from the hellhound_examples repository. Just issue the following command in the repository root directory.

Ok let’s jump into the code.

Run the example code and then use below code in your browser console to test the echo server.

After running the above code you’re going to see "hello world" on your js console.

In this example data flow from webserver component to the output component and back to webserver component as the input and webserver component send its input to the client side application using the ws connection.

Callbacks are a useful building block, but they’re a painful way to create asynchronous workflows. In practice, no one should ever use on-realized.

Instead, they should use manifold.deferred/chain, which chains together callbacks, left to right:

chain returns a deferred representing the return value of the right-most callback. If any of the functions returns a deferred or a value that can be coerced into a deferred, the chain will be paused until the deferred yields a value.

Values that can be coerced into a deferred include Clojure futures, Java futures, and Clojure promises.

If any stage in chain throws an exception or returns a deferred that yields an error, all subsequent stages are skipped, and the deferred returned by chain yields that same error. To handle these cases, you can use manifold.deferred/catch:

Using the → threading operator, chain and catch can be easily and arbitrarily composed.

To combine multiple deferrable values into a single deferred that yields all their results, we can use manifold.deferred/zip:

Finally, we can use manifold.deferred/timeout! to register a timeout on the deferred which will yield either a specified timeout value or a TimeoutException if the deferred is not realized within n milliseconds.

Note that if a timeout is placed on a deferred returned by chain, the timeout elapsing will prevent any further stages from being executed.

Let’s say that we have two services which provide us numbers, and want to get their sum. By using zip and chain together, this is relatively straightforward:

However, this isn’t a very direct expression of what we’re doing. For more complex relationships between deferred values, our code will become even more difficult to understand. In these cases, it’s often best to use let-flow.

In let-flow, we can treat deferred values as if they’re realized. This is only true of values declared within or closed over by let-flow, however. So we can do this:

but not this:

In this example, c is declared within a normal let binding, and as such we can’t treat it as if it were realized.

It can be helpful to think of let-flow as similar to Prismatic’s Graph library, except that the dependencies between values are inferred from the code, rather than explicitly specified. Comparisons to core.async’s goroutines are less accurate, since let-flow allows for concurrent execution of independent paths within the bindings, whereas operations within a goroutine are inherently sequential.

Both deferreds and streams allow for custom execution models to be specified. To learn more, //aleph.io/docs/execution.md[go here].

The simplest thing we can do a stream is consumed every message that comes into it:

However, we can also create derivative streams using operators analogous to Clojure’s sequence operators, a full list of which [can be found here](http://ideolalia.com/manifold/):

Here, we’ve mapped inc over a stream, transforming from a sequence to a stream and then back to a sequence for the sake of a concise example. Note that calling manifold.stream/map on a sequence will automatically call →source, so we can actually omit that, leaving just:

Since streams are not immutable, in order to treat it as a sequence we must do an explicit transformation via stream→seq:

Note that we can create multiple derived streams from the same source:

Here, we create a source stream s, and map inc and dec over it. When we put our message into s it immediately is accepted, since a and b are downstream. All messages put into s will be propagated into both a and b.

If s is closed, both a and b will be closed, as will any other downstream sources we’ve created. Likewise, if everything downstream of s is closed, s will also be closed. This is almost always desirable, as failing to do this will simply cause s to exert back pressure on everything upstream of it. However, If we wish to avoid this behavior, we can create a (permanent-stream), which cannot be closed.

For any Clojure operation that doesn’t have an equivalent in manifold.stream, we can use manifold.stream/transform with a transducer:

There’s also (periodically period f), which behaves like (repeatedly f), but will emit the result of (f) every period milliseconds.

Having created an event source through composition of operators, we will often want to feed all messages into a sink. This can be accomplished via connect:

Again, we see that our message is immediately accepted into a, and can be read from b. We may also pass an options map into connect, with any of the following keys:

Upon connecting two streams, we can inspect any of the streams using description, and follow the flow of data using downstream:

We can recursively apply downstream to traverse the entire topology of our streams. This can be a powerful way to reason about the structure of our running processes, but sometimes we want to change the message from the source before it’s placed into the sink. For this, we can use connect-via:

Note that connect-via takes an argument between the source and sink, which is a single-argument callback. This callback will be invoked with messages from the source, under the assumption that they will be propagated to the sink. This is the underlying mechanism for map, filter, and other stream operators; it allows us to create complex operations that are visible via downstream:

Each element returned by downstream is a 2-tuple, the first element describing the connection, and the second element describing the stream it’s feeding into.

The value returned by the callback for connect-via provides backpressure - if a deferred value is returned, further messages will not be passed in until the deferred value is realized.

We saw above that if we attempt to put a message into a stream, it won’t succeed until the value is taken out. This is because the default stream has no buffer; it simply conveys messages from producers to consumers. If we want to create a stream with a buffer, we can simply call (stream buffer-size). We can also call (buffer size stream) to create a buffer downstream of an existing stream.

We may also call (buffer metric limit stream), if we don’t want to measure our buffer’s size in messages. If, for instance, each message is a collection, we could use count as our metric, and set limit to whatever we want the maximum aggregate count to be.

To limit the rate of messages from a stream, we can use (throttle max-rate stream).

Manifold provides a simple publish/subscribe mechanism in the manifold.bus namespace. To create an event bus, we can use (event-bus). To publish to a particular topic on that bus, we use (publish! bus topic msg). To get a stream representing all messages on a topic, we can call (subscribe bus topic).

Calls to publish! will return a deferred that won’t be realized until all streams have accepted the message. By default, all streams returned by subscribe are unbuffered, but we can change this by providing a stream-generator to event-bus, such as (event-bus #(stream 1e3)). A short example of how event-bus can be used in concert with the buffering and flow control mechanisms [can be found here](https://youtu.be/1bNOO3xxMc0?t=1887). n