powerman/narada repository overview

⁠NAME

Narada - framework for ease deploy and support microservice projects

⁠VERSION

This document describes Narada version v2.3.8

⁠SYNOPSIS

#--- Create new project
~ $ narada-new my_proj
... New project will be created using template (from git repo).

#--- Develop project as usually, until it's ready to run
~ $ cd my_proj
~/my_proj $
... Now you in project's source dir with git repo.
... You can develop this project in any way and language.

#--- Make release and deploy it to check how it works
... NOTE: This operation is usually automated by script(s) provided
... by "project template" used while creating new project.
... Create file 0.1.0.migrate with instructions how to migrate between
... empty directory and version 0.1.0 of your project plus any related
... 0.1.0.{patch,tgz,etc.} files if you need them.
... Copy these files into .release/ subdirectory of directory where
... you want to deploy this version of project, for ex. in _live/.
~/my_proj $ cp 0.1.0.* _live/.release/
~/my_proj $ cd _live
~/my_proj/_live $ narada-install 0.1.0
... Now you in project's deploy directory, with config files, logs,
... running services, data files, etc. and you can check how it works.

#--- Upgrade or downgrade project
... Repeat same steps to prepare 0.2.0.migrate and save into .release/.
~/my_proj/_live $ narada-install 0.2.0
... You can downgrade if something goes wrong.
~/my_proj/_live $ narada-install 0.1.0

#--- You may need to update project's external configuration/data
~/my_proj/_live $ narada-setup-cron
~/my_proj/_live $ narada-setup-mysql
~/my_proj/_live $ narada-setup-qmail
~/my_proj/_live $ narada-start-services

#--- You may need to backup this version and restore another one
~/my_proj/_live $ narada-backup
~/my_proj/_live $ cp .backup/full.tar .backup/full-0.2.0.tar
~/my_proj/_live $ narada-restore .backup/full-0.1.0.tar

#--- You may need to lock project while manual maintenance
~/my_proj/_live $ narada-lock-exclusive
[LOCKED] ~/my_proj/_live $
... Now all project applications will be blocked on next attempt
... to read/write any file or database, so you can safely change
... project files or databases, etc.
[LOCKED] ~/my_proj/_live $ exit
~/my_proj/_live $
... Now project applications will be unblocked and continue to work.

#--- Manage project
... View/monitor project's log.
~/my_proj/_live $ narada-viewlog
... Get console access to project's database.
~/my_proj/_live $ narada-mysql

#--- Stop and uninstall this project.
... Cleanup related external configurations.
~/my_proj/_live $ narada-setup-cron --clean
~/my_proj/_live $ narada-setup-mysql --clean
~/my_proj/_live $ narada-setup-qmail --clean
... Kill all related background processes.
~/my_proj/_live $ narada-shutdown-services
~/my_proj/_live $ narada-bg-killall
... Then just remove it.
~/my_proj/_live $ cd ..
~/my_proj $ rm -rf _live/

⁠DESCRIPTION

Narada was designed for ease development, deploy and support for medium/big server-side project or large amount of small projects (used in microservice architecture). It's a framework which define the way to manage your project (which doesn't really depend on your project's nature), and doesn't restrict your project's implementation in any way. With Narada you can create any projects using any programming languages as long as your project conform to Narada interface and work in *NIX.

Typical example of project which wins a lot when managed by Narada is web site backend or network service, which consists of several applications (even written in different programming languages) with different entry points (HTTP, RPC, cron, email), which all should have common runtime environment, logs, etc.

In short, Narada dictates where your project should keep and how it should work with: config files, logs, locks, temporary and persistent data files, required external configuration for cron, qmail, databases. All of this is called "Narada interface", and if your project conform to it then you can also use a lot of handy command-line tools provided by Narada to deploy, backup and manage your project.

There are few helper Narada::* modules for Perl which helps you to conform Narada interface in your Perl modules/scripts; for other languages you may want to create similar helpers, but this isn't required - Narada interface is simple and can be easily conformed without using special helpers (it was designed that way to make it ease to conform to Narada interface even in shell scripts).

To use Narada you'll also need to learn format of .migrate files used to describe project's upgrade/downgrade operations (see "SYNTAX" in App::migrate⁠) and choose which "project template" to use for your new project (for ex. see default template narada-base⁠; you can modify it with plugins or develop your own template from scratch).

⁠Main features

• Templates and plugins for project source.

  You can choose one of existing templates when starting new project, and even modify some of them using plugins - or made your own templates or plugins if you often develop similar projects. Both templates and plugins are implemented by merging remote repositories, so if they will change after you've started your project you can easily update it by fetching and merging them again.

• You can continue using your favorite workflow.

  While narada-new⁠ tool and existing project templates/plugins use Git, you may opt out and don't use them. This is only Narada tool which do something with your project's source directory (all other tools work only with project's deploy directory) and it's completely optional. You can create new project using Mercurial or even without using any VCS, and use any workflow you like. All you need is produce .migrate and related files as result of releasing every version of your project, including ephemeral development versions which you may release many times per day.

• Provide file/directory structure for deployed project.

  Some choices are already made for you: where and how to store project configuration, temporary and persistent data files, logs, locks, backups, current version number. This file/directory structure was designed to make it ease to use from any programming language (including shell scripts) and provide several valuable features like reliable upgrades and backups.

• Reliable project upgrade and downgrade.

  Narada use .migrate files to describe operations needed to upgrade and (required!) downgrade project. While usual development process you'll run these operations each time you wanna check how your changes work, i.e. this will happens dozens (if not hundreds) of times while you're developing next version, so both upgrade and downgrade operations are usually guaranteed to be well-tested before releasing each next version.

• Different project installations have different configurations.

  Changes in your local configuration won't be occasionally sent with next update to production server. And you won't have any issues because of config files added into your repo with project sources because in Narada projects config files exists only in project's deploy directory.

• Ease project setup after installation/update.

  Narada provide tools to update project's external setup (cron tasks, qmail handlers for incoming emails, MySQL scheme) according to current project's configuration.

• Reliable services.

  Run your FastCGI or RPC daemons with guaranteed restart after crash. Narada project may have own services, always running in background. By default we use runit⁠ for supervising project's services, but other similar supervisors like daemontools or s6 also can be used.

• All project's applications have common log(s).

  When your project consists of many applications/scripts or run many processes of same application it's important to have single common log for all of them. To implement this each Narada project usually run own log service. By default we use socklog⁠ (syslog-compatible daemon) to receive logs from all project's applications and svlogd tool from runit⁠ to manage logs (rotate, filter, group records in separate files, etc.).

• All project's applications have common lock.

  This guarantee live project's consistency while backup, update or manual maintenance. NOTE: While Narada provide and use this lock file in it's own tools it's your responsibility to always get shared lock on that lock file before doing read/write of any project's files or databases - if you won't do this you'll break mentioned above consistency guarantee.

• Consistent and fast project backup.

  Narada interface include shared/exclusive project locking, which let us guarantee backup consistency between project files and databases.

  Narada backup tool support incremental backups both for files and database, which makes it possible to backup most projects in few seconds - your website visitors won't even notice your daily/hourly backup!

• Backward compatibility.

  Whenever possible, projects created using previous Narada versions will be supported by latest Narada (but newly added tools may not work with such an old projects).

⁠Important changes since previous versions

⁠Narada 1.x

Narada 1 was created when Git and Mercurial wasn't exists yet, so it doesn't use repository for project sources. Also it doesn't separate project's source vs deploy directories and run project's applications in directory with it sources - this was perfectly fine for projects developed in script languages like Perl without using VCS. The narada-new tool used to create Narada 1 projects is still available but it was renamed to narada-new-1⁠.

Also Narada 1 provide several tools used to generate, email/upload and apply patches for project's directory - this was the way to both deploy new version to server and distribute changes to other developers in team. These tools has no use in current Narada, but they still exists for compatibility with Narada 1 projects: narada-diff⁠, narada-release⁠, narada-patch-remote⁠, narada-patch-send⁠, narada-patch-pull⁠, narada-patch⁠.

Files&directories structure also was changed since Narada 1.x, see Changes⁠ in version 2.0.0 for more details.

This documentation describe current Narada, check documentation for latest Narada 1⁠ if you're still using Narada 1 projects.

⁠EXAMPLE

Create new project:

~ $ narada-new hello_world
... a lot of Git output skipped
~ $ cd hello_world/
~/hello_world $ ls -AF
build*  deploy*  doc/  .git/  .gitignore  migrate  release*  t/

This is project's source directory initialized using default template. It provide basic migrate files with all commands needed to create all files/directories needed by every deployed Narada project plus few scripts: ./release for releasing new versions (both development ephemeral versions and tagged final versions) and ./deploy for installing them into _live/ subdirectory.

Cool, we already can release something!

~/hello_world $ ./release
t/build/migrate.t .. 1/1 # Checking migrate
t/build/migrate.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( ... )
Result: PASS
~/hello_world $ ls -AF
build*  deploy*  doc/  .git/  .gitignore  migrate  release*  .release/
t/  VERSION
~/hello_world $ ls -AF .release/
0.0.0+b4ff31c.migrate  0.0.0+b4ff31c.patch
~/hello_world $ cat VERSION
0.0.0+b4ff31c

As you see, your project get unique ephemeral version number in file VERSION and this version was released as two files in .release/. Now, let's deploy it!

~/hello_world $ ./deploy
Loading .release/0.0.0+b4ff31c.migrate
... a lot of executed commands output skipped
Migration to 0.0.0+b4ff31c completed
~/hello_world $ ls -AF
build*  deploy*  doc/  .git/  .gitignore  _live/  migrate  release*
.release/  t/  VERSION
~/hello_world $ ls -AF _live/
.backup/  config/  doc/  .lock  .release/  t/  tmp/  var/  VERSION

The _live/ is project's deploy directory. There are may be many of them, even on same development machine in case you develop in several branches and wanna have each branch deployed to separate directory, or just wanna install simultaneously old and new versions of project and compare how they work.

Okay, we get Narada files&directories structure in _live/, but it doesn't have much use for now. Let's add something to our project.

~/hello_world $ cat >hello <<EOF
> #!/usr/bin/perl
> print "Hello, World!\n";
> EOF
~/hello_world $ chmod +x hello
~/hello_world $ git add hello
~/hello_world $ ./release && ./deploy
t/build/migrate.t .. 1/1 # Checking migrate
t/build/migrate.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( ... )
Result: PASS
Loading .release/0.0.0+b4ff31c.dirty-1428492362.migrate
Loading .release/0.0.0+b4ff31c.migrate
Backuping to .backup/full-0.0.0+b4ff31c.tar
...
Migration to 0.0.0 completed
...
Migration to 0.0.0+b4ff31c.dirty-1428492362 completed

What just happens? New ephemeral version "0.0.0+b4ff31c.dirty-1428492362" was released (it have such a name because we didn't committed our changes to the repo yet); then previous version "0.0.0+b4ff31c" installed in _live/ was saved in the backup and downgraded to version "0.0.0" (which is initial version meaning "empty directory"); then that empty directory was upgraded to current version "0.0.0+b4ff31c.dirty-1428492362". Look:

~/hello_world $ cat VERSION
0.0.0+b4ff31c.dirty-1428492362
~/hello_world $ ls -AF .release/
0.0.0+b4ff31c.dirty-1428492362.migrate  0.0.0+b4ff31c.migrate
0.0.0+b4ff31c.dirty-1428492362.patch    0.0.0+b4ff31c.patch
~/hello_world $ ls -AF _live/
.backup/  config/  doc/  hello*  .lock  .release/  t/  tmp/  var/
VERSION
~/hello_world $ cd _live/
~/hello_world/_live $ ./hello
Hello, World!
~/hello_world/_live $ cd -
~/hello_world $

Next, let's start using some Narada features, like config files. We'll also add migration operation to migrate file to create new config file on upgrading to this version and remove it on downgrading from this version.

~/hello_world $ echo 'add_config my_name Powerman' >> migrate
~/hello_world $ cat >hello <<EOF
> #!/usr/bin/perl
> use Narada::Config qw( get_config_line );
> printf "Hello, %s!\n", get_config_line('my_name');
> EOF
~/hello_world $ ./release && ./deploy
...
Loading .release/0.0.0+b4ff31c.dirty-1428493197.migrate
Loading .release/0.0.0+b4ff31c.dirty-1428492362.migrate
Backuping to .backup/full-0.0.0+b4ff31c.dirty-1428492362.tar
...
Migration to 0.0.0 completed
...
Migration to 0.0.0+b4ff31c.dirty-1428493197 completed
~/hello_world $ cd _live/
~/hello_world/_live $ ./hello
Hello, Powerman!

In deploy directory we can safely modify config or data files - these changes will affect only project deployed in this directory.

~/hello_world/_live $ ls -AF config/
backup/  crontab/  log/  my_name  mysql/  qmail/
~/hello_world/_live $ echo Anonymous > config/my_name
~/hello_world/_live $ ./hello
Hello, Anonymous!
~/hello_world/_live $ cd -
~/hello_world $

Now, let's release current version tagged with own version number.

~/hello_world $ git add migrate
~/hello_world $ git commit -m 'add hello'
~/hello_world $ ./release --minor
t/build/migrate.t .. 1/1 # Checking migrate
t/build/migrate.t .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( ... )
Result: PASS
[master 8338faa] Release 0.1.0
 1 file changed, 4 insertions(+)
~/hello_world $ cat VERSION
0.1.0
~/hello_world $ ls -AF .release/
0.0.0+b4ff31c.dirty-1428492362.migrate  0.0.0+b4ff31c.migrate
0.0.0+b4ff31c.dirty-1428492362.patch    0.0.0+b4ff31c.patch
0.0.0+b4ff31c.dirty-1428493197.migrate  0.1.0.migrate
0.0.0+b4ff31c.dirty-1428493197.patch    0.1.0.patch
~/hello_world $

And deploy it on server:

~/hello_world $ ssh localhost 'mkdir -p hello_project/.release'
~/hello_world $ scp .release/0.1.0.* localhost:hello_project/.release/
0.1.0.migrate        100% 7105     6.9KB/s   6.9KB/s   00:00
0.1.0.patch          100% 5223     5.1KB/s   6.9KB/s   00:00
~/hello_world $ ssh localhost
~ $ cd hello_project/
~/hello_project $ ls -AF
.release/
~/hello_project $ ls -AF .release/
0.1.0.migrate  0.1.0.patch
~/hello_project $ narada-install 0.1.0
Loading .release/0.1.0.migrate
...
Migration to 0.1.0 completed
~/hello_project $ ls -AF
.backup/  config/  doc/  hello*  .lock  .release/  t/  tmp/  var/
VERSION
~/hello_project $ ./hello
Hello, Powerman!
~/hello_project $

Finally, let's cleanup and uninstall all projects.

Template used to create this project include some basic cron configuration to make daily project backups. And these settings was already added to your user's crontab while installing the project. So, before removing project directories we should remove this cron setup.

~/hello_project $ narada-setup-cron --clean
~/hello_project $ cd ..
~ $ rm -rf hello_project/
~ $ exit
~/hello_world $ cd _live/
~/hello_world/_live $ narada-setup-cron --clean
~/hello_world/_live $ cd ../..
~ $ rm -rf hello_world/

⁠INTERFACE

The "Narada interface" is described here files&directories structure for project's deploy directory and some conventions about how they should be used. Your project must conform to this interface.

⁠Directory types

There are two types of "root" directories in your project: source directory and deploy directory.

The project's source directory isn't part of Narada interface. Only tool which work with it is narada-new⁠ (which helps you to create new project), but you're not required to use it, and even if you use it no Narada tools will touch your project's source directory after it will be created.

The project's deploy directory is the one where all files&directories defined by Narada interface should exists. Also it's the directory where you should run all Narada tools and your project's applications - this is required to allow them to find all Narada files&directories using paths relative to current directory.

It's ok to have many source directories (as repo clones for your team) and to have many deploy directories (as different installations on same or different computers) - for ex. it may be very useful to deploy different versions from different Git branches to different deploy directories on same development machine.

⁠Overview of deploy directory structure

Project templates often include typical directories like doc/ or t/ but they isn't part of Narada interface and thus you're free to rename or remove them if you like.

• .release/

  Contain .migrate and related files used to upgrade and downgrade project while migrating to another version.

• .backup/

  Contain project backups. You can create and manage them yourself, but they also will be automatically created before migrating to another version and may be automatically used when only way to downgrade project is restore previous version from backup. In general it's safe to remove backups when you like, but absence of some backups may make it impossible to downgrade to some previous version.

• VERSION

  Contain current project version. Required for upgrade and downgrade, and will be automatically updated after migration or restoring from backup. May be useful for your applications (read-only).

• .lock*

  Several lock files used mostly internally by Narada tools, except for the .lock file which should be shared-locked by all your applications while they read or write any project's files or databases.

• config/

  Project's configuration (both defined by Narada interface and custom settings of your project). May differ in different project's deploy directories. While it's possible to modify configuration in all deploy directories while project migration, usually most of config files modified either manually or by your applications in one deploy directory.

• service/

  Used to setup project's services (daemons) and run them using service supervisor (runit, daemontools, etc.). Most projects usually have just one (log) or a couple (log and fastcgi/http/rpc) services.

• tmp/

  Used for temporary data files. Contents of this directory won't be included in backups.

• var/

  Used for persistent data files.

⁠Deploy

In "EXAMPLE"⁠ above you've seen scripts ./release, ./deploy, files migrate and VERSION, directories .release/ and _live/ while working with Narada, but all of them was in project's source directory and isn't part of Narada interface! All of them was provided by used project template, and different templates may implement these tasks in different ways - read documentation for chosen project template. Also you're free to modify these paths and scripts in any way - template provide only starting point, but it's your project's sources and you have freedom to do anything you like.

What is actually part of Narada interface is result of running these ./release && ./deploy scripts: file .release/<version>.migrate and optional related files (usually named .release/<version>.patch or .release/<version>.tgz) in project's deploy directory. No matter how you develop, build, compile, release and copy/upload new version to deploy directory, the final result should be such file/files.

• .release/<version>.migrate

  These files must contain upgrade and downgrade operations between ve