Rob Ward's Log

Valid XHTML 1.0 Strict Valid CSS!


I originally started this log with good intentions of keeping it up to date and creating a few nice projects, unfortunately existence(mainly work; 45+ hour weeks and 20+ hours travelling really makes finding time difficult) got in the way and so nothing really happened and I stopped updating the log.

After 16 months I have decided that while I don't really have time for any large projects I should still document things that I do work on. A number of things have happened in the last 16 months that I haven't documented which I wish I had, mainly as a reference for myself but it may help others. In the last 16 months I have also changed jobs, due to decreased travel time I now have more free time available so am doing more interesting things that I should document.

So hopefully there will be more updates in the coming months, I am also aiming to update some of the other pages as most still have no content.

Many months later with no updates - Sigh!!!


Over the last couple of days I have found several hours to play with different tools, insomnia does have a few advantages....

One of the things that I spend some time on was re-spinning the phram changes that I mentioned in the entry the other day[0]. I have done some testing and it still seems to be working fine. I want to do one more test in both built in and built as a module, I will then send the new version to the kernel mailing list. While I would still like a nicer way of doing it than I currently have I am more happy with the solution now the size isn't hard coded. Anyone that is that is interested in the current version a copy is here[1].

I also spent a bit more time on musl compilers, I found another project[2] that is designed for building cross compilers, this one is specifically designed for musl compilers and worked first time. I think that at least to start with I will use this over crosstool-ng just for simplicities sake. I'm hoping over the weekend to get an x86 and either a mips or an arm compiler build and the kernel/toybox/busybox compiled and tested.

[0] -
[1] -
[2] -


Okay, several days have passed including a weekend since I did an update. I didn't get as much time over the weekend as I wanted so no great leap forward on any project.

I have done a number of little things that add up though.

Firstly I have respun/created my websites front page and some content pages. I'm still keeping the website as simple as possible, I have done CSS and Javascript heavy "fancy" websites in the past with their own custom content management systems etc and quite frankly I prefer to keep it simple, clean and easy to read. I need to do a bit of clean up on the HTML as it has got slightly untidy while I was working on it but that is a simple enough task. I store my website in a git repository in order to keep track of what I change, I pushed the latest version last night and put a copy on the server. A number of the pages currently have no content on but I will fill them up slowly.

I have added a page called Linux in the Browser, this is probably slightly premature but the intention of this page is to have a play with the v86[0] javascript library and put a custom slimmed down Linux distro on my website utilising it. I don't intend the system to be fast or used for many tasks, I would however like one that has a large selection of command line tools and more importantly the man pages so that I can use it for reference. Plus I haven't used the library before so it sounds like fun... Hopefully I will find some time to have a play and see how well it works.

I also spent a chunk of my weekend fixing, or rather adding some functionality into the kernel. For some reason the ability to disable the /dev/mem device is not present. You can however disable the /dev/kmem device, not sure why one and not the other. This seems wrong to me as on smaller and more secure systems I would much rather not have the ability for the contents of RAM to be read via a dev node. So I spend an hour adding this support in, the patch that I have so far is available in my downloads[1]. I haven't completed testing yet, I need to do 10 builds with different configs due to the defines I am modifying and check the behaviour of each is correct. I have done most and I am relatively happy that it works fine but there are a few more I want to check.

For the record the list of build combos I need are:


I haven't pushed this change up to the kernel mailing list as of yet, the Kernel team is currently in the middle of a release cycle, 3.8-rc3 was released a couple of days ago, so the merge window is still a number of weeks away meaning I have plenty of time.

With regards to kernel changes I also have another waiting in review[2] to fix an issue in the phram mtd code to allow the creation of multiple phram devices on boot via the kernel command line. Unfortunately the fix for phram is slightly horrible, the code in question runs prior to the ability for kmalloc to be used meaning that to store multiple options from the command line you can't use the usual linked list or dynamically sized array that can grow to contain as many elements as you need. Instead I had to make it used a fixed size array, I really don't like this but I can't think of a better way of doing it at the moment, at least not one that is sane. I'm kinda hoping that one of the maintainer will turn around, call me stupid and point me to some area of the kernel with a nice solution to the problem that I am not aware of, if not I think I will probably respin the change to include a Kconfig option to allow the size of the array to be defined. Currently I have hard coded the size to be char fred[64][104], this will likely be overkill for nearly everyone. If I replace the 64 with a option this can default to 1 and there will be no difference to the current behaviour but it will have support in for increasing the max number of phram devices based on given use cases. This will also allow you to keep the size of the kernel to a minimum which should keep people happy.

I also started looking at compilers and the best way to create a custom compiler without going to the effort(at least at the moment) of building one manually. I have used Crosstool-ng and made it create a GCC musl compiler(untested). This was more tricky than I would have liked and I had do disable certain features and choose an older version of GCC in order to get it to compile. I will need to test this and then I will try and use a similar config to create a cross compiler and see how that goes.

One other thing that I have done is look at how and if it is a good idea to use a fakechroot for building all of the source. I am not sure on this either way as of yet... Would be interesting but would mean that the build system I need to create would have to be split into two parts, one part would run to construct and build the required host tools in order to bootstrap the fakechroot and then the second would have to build the rest of the the system.

[0] -
[1] -
[2] -


Making progress slowly, not getting much time at the moment, I do however commute a long way so have managed to spend a bit of time on my less busy trains getting a few bits implemented.

So far I have added the Linux stable repo into my project as a submodule. I am using the upstream version from as I don't currently need to modify any of the kernels source. If that changes I will switch to using my own version hosted on GitHub.

As well as adding the Linux kernel source I have also added a first version of a makefile. Currently it is slightly messy and only defines a few variables and a couple of targets. The main target is for building the Linux kernel.

Currently the way I build the kernel is hacky but it works as a first stage. The makefile itself is in the git repo already but the following in the set of commands that are in the make target:

$(QUIET)cd $(LINUX_SRC_DIR) && $(MAKE) x86_64_defconfig
$(QUIET)cd $(LINUX_SRC_DIR) && $(MAKE) --jobs=4 modules
$(QUIET)cd $(LINUX_SRC_DIR) && $(MAKE) --jobs=4 modules_install INSTALL_MOD_PATH=$(INITRAMFS_DIR)
# Now setup the initramfs
$(QUIET)cd $(LINUX_SRC_DIR) && $(MAKE) --jobs=4

I have chosen to use the default x86_64_defconfig, this keeps the current implementation very simple. I will however need to implement the use of custom kernel configs and the ability to build different architecture in the not too distant future.

What the current build target does do however is build the kernel modules and install them in a location that can then be included in the initramfs. I have also put some sed commands in place that setup the source location for the initramfs and the UID and GID. The sed commands are rather evil but they work for now.

The CONFIG_INITRAMFS_SOURCE is used to give a list of locations of files to include in the initramfs. This can either be a config file or a directory. If it is a directory then the contents are included in the initramfs. In my current implementation I pass both a config file and a directory. The directory that I pass is the INITRAMFS_DIR, this is where the kernel modules are installed after building and where I will install any of the other utilities that I need in the initramfs going forward. The config file is a plain text file that is used to specify directories or device nodes that should be created. The current config file I am using is very simple and creates the directories /dev, /proc, /sys and the device node /dev/console. My current config is in the git repo but looks like this:

dir /dev 755 0 0
nod /dev/console 644 0 0 c 5 1
dir /proc 755 0 0
dir /sys 755 0 0

The CONFIG_INITRAMFS_ROOT_{UID/GID} are used to tell the kernels build system to map the files user/group permissions from the given UID/GID to be owned by the root user.

You will notice looking at the build commands above that I am calling make with --jobs=4 in order to build multi-threaded. I chose 4 simply because I was on my laptop with 4 CPU's. A future enhancement will make the number of jobs dynamic depending on the number of cores the machine building has, well unless I add distcc into the mix.

So where does that leave me at the minute? Well if I run the make command a kernel is successfully built. The end result of the build is:

Kernel: arch/x86/boot/bzImage is ready (#7)

This bzImage file is the kernel + initramfs image that we can run.

To run the image I am currently using qemu, the command I am using to run it is:

qemu-system-x86_64 -kernel src/Linux-stable/arch/x86/boot/bzImage -append "console=ttyS0" -serial stdio

When run the kernel boots up until the point where it panics and explodes horribly:

[ 2.589339] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.590230] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.1 #7
[ 2.590230] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 2.590230] 0000000000008001 ffff8800070a3dd8 ffffffff81805696 ffffffff81baa678
[ 2.590230] ffff8800070a3e50 ffffffff81801e78 0000000000000010 ffff8800070a3e60
[ 2.590230] ffff8800070a3e00 0000000800000000 ffff8800070a3e70 0000000000000014
[ 2.590230] Call Trace:
[ 2.590230] [<ffffffff81805696>] dump_stack+0x45/0x56
[ 2.590230] [<ffffffff81801e78>] panic+0xbd/0x1e1
[ 2.590230] [<ffffffff81ef143b>] mount_block_root+0x183/0x221
[ 2.590230] [<ffffffff81ef15d3>] mount_root+0xfa/0x103
[ 2.590230] [<ffffffff81ef1718>] prepare_namespace+0x13c/0x174
[ 2.590230] [<ffffffff81ef1171>] kernel_init_freeable+0x1c5/0x1d3
[ 2.590230] [<ffffffff81ef08f2>] ? do_early_param+0x8a/0x8a
[ 2.590230] [<ffffffff817feb40>] ? rest_init+0x80/0x80
[ 2.590230] [<ffffffff817feb49>] kernel_init+0x9/0xf0
[ 2.590230] [<ffffffff8180f22c>] ret_from_fork+0x7c/0xb0
[ 2.590230] [<ffffffff817feb40>] ? rest_init+0x80/0x80
[ 2.590230] Kernel Offset: 0x0 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffff9fffffff)
[ 2.590230] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 2.590230] general protection fault: fff2 [#1] SMP
[ 2.590230] Modules linked in:
[ 2.590230] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.17.1 #7
[ 2.590230] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[ 2.590230] task: ffff880007098000 ti: ffff8800070a0000 task.ti: ffff8800070a0000
[ 2.590230] RIP: 0010:[<ffffffff81801f63>] [<ffffffff81801f63>] panic+0x1a8/0x1e1
[ 2.590230] RSP: 0000:ffff8800070a3de8 EFLAGS: 00000246
[ 2.590230] RAX: 0000000000000057 RBX: ffffffff81baa678 RCX: 00000000000000d4
[ 2.590230] RDX: 0000000000000063 RSI: 0000000000000046 RDI: ffffffff82072cc4
[ 2.590230] RBP: ffff8800070a3e50 R08: 6b6e75206e6f2073 R09: 0000000000000000
[ 2.590230] R10: 6c622d6e776f6e6b R11: 29302c30286b636f R12: 0000000000000000
[ 2.590230] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88000727a000
[ 2.590230] FS: 0000000000000000(0000) GS:ffff880007c00000(0000) knlGS:0000000000000000
[ 2.590230] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2.590230] CR2: 0000000000000000 CR3: 0000000001e14000 CR4: 00000000000006f0
[ 2.590230] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2.590230] DR3: 0000000000000000 DR6: 0000000000000000 DR7: 0000000000000000
[ 2.590230] Stack:
[ 2.590230] 0000000000000010 ffff8800070a3e60 ffff8800070a3e00 0000000800000000
[ 2.590230] ffff8800070a3e70 0000000000000014 0000000000000000 0000000800000000
[ 2.590230] 00000008ffffffff ffff88000727a000 0000000000008001 0000000000008001
[ 2.590230] Call Trace:
[ 2.590230] [<ffffffff81ef143b>] mount_block_root+0x183/0x221
[ 2.590230] [<ffffffff81ef15d3>] mount_root+0xfa/0x103
[ 2.590230] [<ffffffff81ef1718>] prepare_namespace+0x13c/0x174
[ 2.590230] [<ffffffff81ef1171>] kernel_init_freeable+0x1c5/0x1d3
[ 2.590230] [<ffffffff81ef08f2>] ? do_early_param+0x8a/0x8a
[ 2.590230] [<ffffffff817feb40>] ? rest_init+0x80/0x80
[ 2.590230] [<ffffffff817feb49>] kernel_init+0x9/0xf0
[ 2.590230] [<ffffffff8180f22c>] ret_from_fork+0x7c/0xb0
[ 2.590230] [<ffffffff817feb40>] ? rest_init+0x80/0x80
[ 2.590230] Code: 39 d8 7f b8 83 3d fd f1 82 00 00 74 05 e8 16 5c 86 ff 48 c7 c6 60 11 03 82 48 c7 c7 d8 7e bb 81 31 c0 e8 77 05 00 00 fb 45 31 e4 <4d> 39 ec 7c 18 41 83 f6 01 44 89 f7 ff 15 ab f1 82 00 49 01 c4
[ 2.590230] RIP [<ffffffff81801f63>] panic+0x1a8/0x1e1
[ 2.590230] RSP <ffff8800070a3de8>
[ 2.590230] ---[ end trace 9dbf8f6652588d07 ]---

Generally when you boot a Linux kernel it will try and run the application /init which is used to setup and configure userspace and run every other application on your system. The build I ran above didn't have an init so once the kernel finishes booting it has nothing to call and dies horribly.

As such my next step was to create a minimal init application to include in the initramfs. The simplest way of creating a init application is a statically linked C application. I have created this and placed in another git repo called microinit[0] that I have added to GitHub. I have then added this as a submodule and added the microinit target in the makefile.

At this point I now have a kernel+initramfs that boots and runs, admittedly all it does at the moment is sit and say "Microinit is running" once a second, but small steps.

My next stage will be to get an actual command line up and running. I have usually used Busybox[1], however for what I am doing as space/speed isn't really the point I could use the full utilities as ship with most distros. The other choice is the use a newer package called Toybox[2], I have never used this before so I may have to give it ago, I know it isn't finished but it may do what I need at the moment.

I also have to start thinking about toolchains and how to manage them. Currently I think I will probably build a Toolchain(or a few) and store them as binaries rather than building a toolchain every time.

One other think I am considering doing is making all of this build in a fakechroot environment, I have used this a few times for compiling software and it does make cross compiling easier for certain uncooperative packages, I haven't built a full system using it though. Again this would be something new to try so I think I will have ago and see what happens. It does mean that I will need to build a number of the host tools, or take copies of them into the fakechroot along with the compiler that I am using to cross compile with. This adds complexity but would make the system more self contained.

[0] -
[1] -
[2] -


So, the first stage for me to build a custom Linux distro is to sort out repos for the build system and packages I want to build.

I previously decided I was going to use github to host this work, at least initially. As such my plan is to create a single repo to store my build system. I will also have a separate repo for storing each of the packages I am going to build, at least I will for any that are not already available in a git repo or where I need the ability to modify them.

I generally use the git-repo[0] tool developed by Google for the android project when I want to manage multiple git repos. Git-repo works very well and gives you a lot of control, especially when sharing code between multiple teams.

I have however decided that I am going to use git submodules[1] instead, I have used these in the past and they should do what I need at the moment, I can always change later easily enough if I need to.

So to start with I will be creating a repo that will manage the submodules and also hold the top level build system and documentation. Below this I will place all the source that I am building as separate repos.

So that is it, I have created the repo[2] and all I have to do is start adding code and building up a system.

Easy right....

Well actually the first few stages are quite easy, my first task will be to add some build commands for the kernel and get it building and running in QEMU. I will do the first stage with a initramfs only(I will cover what this is later) and I will add a microscopic statically linked init system into the mix as well to prevent the kernel rebooting straight away. At this point I will need to decide on a plan for what I want to achieve first and how...

[0] -
[1] -


Okay folks, after several weeks(months(years???)) of thinking about it I have decided to create my own custom Linux[0] distro.

At this point I should probably clarify that this is not the first time that I have made my own custom distro. I work developing embedded Linux devices and as part of this I have developed custom Linux distros or built upon others custom distributions on several occasions. I have also personally built a number of simplified Linux distros in the past, usually for fun and to learn new technologies.

So what is different this time??? Well in the past the distros I have developed have all stopped at embedded device level, usable yes, but of limited functionality usually restricted to command line interfaces or running services i.e. web servers. In short, they have all had limited scope. This time I have decided that I want to create something on a larger scale, with nearly unlimited scope. I am not intending what I create to be of use for one specific purpose but rather as a personal playground where I can test and learn new technologies. In no way do I intend to create a Linux distro that you would download and install on your Grandparents PC. Hell, I don't really expect it to be something that anyone that isn't wanting to learn about or hack a Linux system will be even touching!

So what is the Plan? Well I suppose my plan is to come up with a plan. I do however have certain points already mapped out.

Firstly I am (best intentions) going to try and keep this blog up to date with the progress that I am making, along with anything useful I learn on the way. I am also planning to do as much in the open as possible.

The development will happen using git as my version control system, generally I use Gerrit[1] as my git backend, however at least for now I am indenting to use GitHub[2] to store and share my work in progress. The key plan is to make everything available.

The majority of the work, at least to start with is creating a make system that is able to create a bootable Linux+RootFS. To start with this will all be done using GNU Make[3], however I have several ideas that I would like to explore with regards to make systems so I will likely swap and change this at some point.

I will initially be using QEMU[4] for development but that can(and will) change. I am also aiming to make the system in such a way that it is able to cross compile. Cross compilation generally adds a lot of complexity, however it highlights issues with packages that you are building very early. It also means you end up fixing broken upstream make systems quite often. I must admit at this point that I am slightly fearful of attempting to get Xorg cross compiling, it was painful enough a few years ago when I compiled it natively... but it is just another challenge to overcome.

The system that I am building will need the ability to build itself so that will be one of the goals. I am also going to be using musl[5] as my c library, the primary reason for this is it looks like a nice project and I haven't used it for anything big as of yet. I will however aim to make the build system so it can support others(uClibc[6] and glibc[7] are the obvious ones but who knows).

The other thing I want to get right, and this relates back to potential alternatives to GNU make, is building in parallel as much as possible. When building something like this you should be able to use as many CPU cores as possible, I may even set it up for distcc[8] just for fun.

If I progress far enough I will have to start looking at storing parts of the builds as binaries, however that will be a long way down the road.

So that is it, the part of a plan that I have. As I figure more out I will update here, along with adding links to any repos with source in that I need.

[0] -
[1] -
[2] -
[3] -
[4] -
[5] -
[6] -
[7] -
[8] -