NASLite Network Attached Storage

So both UPNP and DAAP at peek (under 50M) are well below the threshold of your RAM. Another thing to consider, the kernel is killing DAAP, UPNP and MDNS in order of priority - lowest first. That rules out problems with the DAAP daemon since it's not the only one failing. My guess would be bad RAM, so please run memtest and if at all possible, leave it running overnight.

I ran the memtest app all night last night and did find errors. It looks like by default it only tests the first stick of memory in the system. I had 2 sticks - a 128 and a 256. Checking the config in the morning showed that only the 128 was mapped to an address range so I assume that the errors must be in the 128. This also means, of course, that the 256 is still untested.

I've removed the 128 and am testing DAAP again now with just the 256. If I still get problems I'll do the memtest again on that but I do need to try and keep this server up as much as I can.

Quick question - if the memory is in good condition should I expect to get no errors at all or are a few errors acceptable?

John

Ok some more testing and I think more useful results.

This time I tested with a single 256Mb memory stick. I had UPNP turned on so 500Mb of swap available. I set the status update rate to 1 minute. DAAP background media indexing - ON, DAAP media indexing type - FAST, UPNP Compliance profile - GENERIC. I saved the configuration and rebooted.

This time I refreshed the process status page every minute and wrote down the memory usage of the DAAP and UPNP processes. Here's what I found:

Minute 1
DAAP 3588Kb
UPNP 32000Kb
0M swap used

Minute 2
DAAP 3828Kb
UPNP 55264Kb
0M swap used

Minute 3
DAAP 533880Kb
UPNP 61568Kb
0M swap used

Minute 4
DAAP 674976Kb
UPNP 2432 Kb
225M swap used

Minute 5
DAAP process dies
UPNP remains running but shows no files available in Windows Media Player

And here's the end of the syslog:

# Jan 13 12:09:50 [2] Adding Swap: 509944k swap-space (priority -1)
# Jan 13 12:14:13 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 13 12:14:13 [4] VM: killing process mt-daapd

I'm starting to think this ISN'T a bad memory problem (although it does seem that initial 128Mb stick was suspect). I've now tested with 2 more 256Mb sticks and in both cases the UPNP service survived but the DAAP process didn't. From the data above it looks like DAAP memory usage does shoot way up just before the process dies. I wish we could get more frequent data than one sample per minute but it seems likely that DAAP really did simply run the machine out of RAM and swap.

John

And here's another test report:

This time I did the same test as above but with 2 sticks of 256Mb RAM for a total of 512Mb RAM and 1003Mb SWAP (according to the status page. Here are the results:

12:35
SWAP usage 0M
DAAP 3192 K
UPNP 30040K

12:37
SWAP usage 0M
DAAP 3828K
UPNP 54080K

12:39
SWAP usage 0M
DAAP 1309512K
UPNP 60576K

12:40
SWAP usage 226M
DAAP 1438524K
UPNP 2592K

12:41
SWAP usage 482M
DAAP 1440924K
UPNP 2592K

12:42
SWAP usage 733M
DAAP 1440312K
UPNP 2592K

12:43
SWAP usage 987M
DAAP 1442004K
UPNP 2432K

12:44
SWAP usage 8M
DAAP 0K
UPNP 2432K
DAAP process dies

Syslog:
# Jan 13 12:35:53 [2] Adding Swap: 1028088k swap-space (priority -1)
# Jan 13 12:43:13 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 13 12:43:13 [4] VM: killing process mt-daapd

So as we can see - with twice as much RAM in the system the DAAP process ran for 9 minutes compared to the previous 5 before it exhausted all the available RAM and SWAP.

It seems pretty clear to me now that the problem we're seeing in this instance is not related to bad RAM. It's simply that the DAAP process is getting too large and running the system out of resources. Is there some way I can configure the system so that DAAP uses less ram at the expense of indexing slower? And will installing M2 onto a HD in the machine create a swap partition that is more than 2xRAM?

John

What you are describing is highly abnormal so I have to say there is a memory leak somewhere. Since no one else is speaking up and since I can't replicate it, I have to say that it's likely isolated to a driver or a combination of drivers associated with your hardware.

Try running with background indexing turned off and see if that makes a difference.

Meanwhile, I'll build a new kernel for you to try. Without a way to replicate what you are experiencing it will be a shot in the dark.

Thanks mate. Would it help to post the whole syslog? There are also some messages during boot about what drivers it's selecting that I could try to scribble down. It's an old Dell Pentium2 system.

John

Sorry to say that turning off background indexing made no difference.

John

Syslog may be helpful.

Also, run memtest on the 256. If that comes up bad too then you may want to test the RAM on another machine also. You may have a bad motherboard. Bad motherboard or bad RAM can lead to data corruption so don't take chances.

I'll try and get another ISO in the next day or so.

Sorry I forgot to post this yesterday. This is a complete Syslog. Hope it provides some useful info.

* Jan 13 14:13:01 [3] klogd started: BusyBox v1.01 (2008.07.26-02:54+0000)
* Jan 13 14:13:01 [4] Linux version 2.4.36.NASLite (root@tzt) (gcc version 3.4.6) #2 Wed Jul 16 21:48:01 EDT 2008
* Jan 13 14:13:01 [2] BIOS-provided physical RAM map:
* Jan 13 14:13:01 [4] BIOS-e820: 0000000000000000 - 000000000009f400 (usable)
* Jan 13 14:13:01 [4] BIOS-e820: 000000000009f400 - 00000000000a0000 (reserved)
* Jan 13 14:13:01 [4] BIOS-e820: 00000000000e7800 - 0000000000100000 (reserved)
* Jan 13 14:13:01 [4] BIOS-e820: 0000000000100000 - 00000000040fdc00 (usable)
* Jan 13 14:13:01 [4] BIOS-e820: 00000000040fdc00 - 00000000040ff800 (ACPI data)
* Jan 13 14:13:01 [4] BIOS-e820: 00000000040ff800 - 00000000040ffc00 (ACPI NVS)
* Jan 13 14:13:01 [4] BIOS-e820: 00000000040ffc00 - 0000000020000000 (usable)
* Jan 13 14:13:01 [4] BIOS-e820: 00000000fffe7800 - 0000000100000000 (reserved)
* Jan 13 14:13:01 [3] 0MB HIGHMEM available.
* Jan 13 14:13:01 [3] 512MB LOWMEM available.
* Jan 13 14:13:01 [4] On node 0 totalpages: 131072
* Jan 13 14:13:01 [4] zone(0): 4096 pages.
* Jan 13 14:13:01 [4] zone(1): 126976 pages.
* Jan 13 14:13:01 [4] zone(2): 0 pages.
* Jan 13 14:13:01 [3] ACPI disabled because your bios is from 98 and too old
* Jan 13 14:13:01 [3] You can enable it with acpi=force
* Jan 13 14:13:01 [4] Kernel command line: rw root=/dev/ram0 initrd=NASLite.02 quiet BOOT_IMAGE=NASLite.01
* Jan 13 14:13:01 [4] No local APIC present or hardware disabled
* Jan 13 14:13:01 [2] Initializing CPU#0
* Jan 13 14:13:01 [4] Detected 348.494 MHz processor.
* Jan 13 14:13:01 [4] Console: colour VGA+ 80x25
* Jan 13 14:13:01 [4] Calibrating delay loop. 694.68 BogoMIPS
* Jan 13 14:13:01 [2] Memory: 509872k/524288k available (2325k kernel code, 14016k reserved, 680k data, 556k init, 0k highmem)
* Jan 13 14:13:01 [2] Dentry cache hash table entries: 65536 (order: 7, 524288 bytes)
* Jan 13 14:13:01 [2] Inode cache hash table entries: 32768 (order: 6, 262144 bytes)
* Jan 13 14:13:01 [2] Mount cache hash table entries: 512 (order: 0, 4096 bytes)
* Jan 13 14:13:01 [2] Buffer cache hash table entries: 32768 (order: 5, 131072 bytes)
* Jan 13 14:13:01 [4] Page-cache hash table entries: 131072 (order: 7, 524288 bytes)
* Jan 13 14:13:01 [2] CPU: L1 I cache: 16K, L1 D cache: 16K
* Jan 13 14:13:01 [2] CPU: L2 cache: 512K
* Jan 13 14:13:01 [2] Intel machine check architecture supported.
* Jan 13 14:13:01 [2] Intel machine check reporting enabled on CPU#0.
* Jan 13 14:13:01 [1] CPU: After generic, caps: 0183f9ff 00000000 00000000 00000000
* Jan 13 14:13:01 [1] CPU: Common caps: 0183f9ff 00000000 00000000 00000000
* Jan 13 14:13:01 [4] CPU: Intel Pentium II (Deschutes) stepping 02
* Jan 13 14:13:01 [2] Enabling fast FPU save and restore. done.
* Jan 13 14:13:01 [2] Checking 'hlt' instruction. OK.
* Jan 13 14:13:01 [4] POSIX conformance testing by UNIFIX
* Jan 13 14:13:01 [4] mtrr: v1.40 (20010327) Richard Gooch (rgooch@atnf.csiro.au)
* Jan 13 14:13:01 [4] mtrr: detected mtrr type: Intel
* Jan 13 14:13:01 [2] ACPI: Subsystem revision 20040326
* Jan 13 14:13:01 [2] ACPI: Interpreter disabled.
* Jan 13 14:13:01 [2] PCI: PCI BIOS revision 2.10 entry at 0xfd9a3, last bus=1
* Jan 13 14:13:01 [2] PCI: Using configuration type 1
* Jan 13 14:13:01 [2] PCI: Probing PCI hardware
* Jan 13 14:13:01 [4] PCI: Probing PCI hardware (bus 00)
* Jan 13 14:13:01 [2] PCI: Using IRQ router PIIX/ICH [8086/7110] at 00:07.0
* Jan 13 14:13:01 [2] Limiting direct PCI/PCI transfers.
* Jan 13 14:13:01 [2] Linux NET4.0 for Linux 2.4
* Jan 13 14:13:01 [2] Based upon Swansea University Computer Society NET3.039
* Jan 13 14:13:01 [4] Initializing RT netlink socket
* Jan 13 14:13:01 [4] Starting kswapd
* Jan 13 14:13:01 [2] Journalled Block Device driver loaded
* Jan 13 14:13:01 [2] devfs: v1.12c (20020818) Richard Gooch (rgooch@atnf.csiro.au)
* Jan 13 14:13:01 [2] devfs: boot_options: 0x1
* Jan 13 14:13:01 [2] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
* Jan 13 14:13:01 [4] pty: 256 Unix98 ptys configured
* Jan 13 14:13:01 [2] Real Time Clock Driver v1.10f
* Jan 13 14:13:01 [2] Floppy drive(s): fd0 is 1.44M
* Jan 13 14:13:01 [2] FDC 0 is a post-1991 82077
* Jan 13 14:13:01 [4] RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize
* Jan 13 14:13:01 [2] loop: loaded (max 8 devices)
* Jan 13 14:13:01 [2] Intel(R) PRO/1000 Network Driver - version 7.3.20-k4
* Jan 13 14:13:01 [2] Copyright (c) 1999-2006 Intel Corporation.
* Jan 13 14:13:01 [2] dgrs: SW=$Id: dgrs.c,v 1.13 2000/06/06 04:07:00 rick Exp $ FW=Build 550 11/16/96 03:45:15
* Jan 13 14:13:01 [4] FW Version=$Version$
* Jan 13 14:13:01 [2] pcnet32.c:v1.30h 06.24.2004 tsbogend@alpha.franken.de
* Jan 13 14:13:01 [2] ThunderLAN driver v1.15
* Jan 13 14:13:01 [2] TLAN: 0 devices installed, PCI: 0 EISA: 0
* Jan 13 14:13:01 [2] dmfe: Davicom DM9xxx net driver, version 1.36.4 (2002-01-17)
* Jan 13 14:13:01 [2] ns83820.c: National Semiconductor DP83820 10/100/1000 driver.
* Jan 13 14:13:01 [4] sk98lin: No adapter found.
* Jan 13 14:13:01 [2] forcedeth.c: Reverse Engineered nForce ethernet driver. Version 0.50.
* Jan 13 14:13:01 [2] 8139cp: 10/100 PCI Ethernet driver v1.2 (Mar 22, 2004)
* Jan 13 14:13:01 [5] 8139cp: pci dev 00:10.0 (id 10ec:8139 rev 10) is not an 8139C+ compatible chip
* Jan 13 14:13:01 [5] 8139cp: Try the "8139too" driver instead.
* Jan 13 14:13:01 [2] 8139too Fast Ethernet driver 0.9.26
* Jan 13 14:13:01 [2] PCI: Found IRQ 11 for device 00:10.0
* Jan 13 14:13:01 [2] PCI: Sharing IRQ 11 with 00:07.2
* Jan 13 14:13:01 [2] eth0: RealTek RTL8139 at 0xe0800000, 00:00:21:f8:15:7a, IRQ 11
* Jan 13 14:13:01 [1] eth0: Identified 8139 chip type 'RTL-8139B'
* Jan 13 14:13:01 [2] -2.4
* Jan 13 14:13:01 [2] ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xx
* Jan 13 14:13:01 [2] PIIX4: IDE controller at PCI slot 00:07.1
* Jan 13 14:13:01 [2] PIIX4: chipset revision 1
* Jan 13 14:13:01 [2] PIIX4: not 100% native mode: will probe irqs later
* Jan 13 14:13:01 [2] ide0: BM-DMA at 0x1420-0x1427, BIOS settings: hda:pio, hdb:DMA
* Jan 13 14:13:01 [2] ide1: BM-DMA at 0x1428-0x142f, BIOS settings: hdc:pio, hdd:DMA
* Jan 13 14:13:01 [4] hda: Maxtor 6L250R0, ATA DISK drive
* Jan 13 14:13:01 [4] hdb: Maxtor 6L250R0, ATA DISK drive
* Jan 13 14:13:01 [4] hdc: NEC CD-ROM DRIVE:28D, ATAPI CD/DVD-ROM drive
* Jan 13 14:13:01 [4] hdd: IBM-DTLA-305040, ATA DISK drive
* Jan 13 14:13:01 [4] ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
* Jan 13 14:13:01 [4] ide1 at 0x170-0x177,0x376 on irq 15
* Jan 13 14:13:01 [4] hda: attached ide-disk driver.
* Jan 13 14:13:01 [4] hda: host protected area => 1
* Jan 13 14:13:01 [2] hda: 490234752 sectors (251000 MB) w/16384KiB Cache, CHS=30515/255/63, UDMA(33)
* Jan 13 14:13:01 [4] hdb: attached ide-disk driver.
* Jan 13 14:13:01 [4] hdb: host protected area => 1
* Jan 13 14:13:01 [2] hdb: 490234752 sectors (251000 MB) w/16384KiB Cache, CHS=30515/255/63, UDMA(33)
* Jan 13 14:13:01 [4] hdd: attached ide-disk driver.
* Jan 13 14:13:01 [4] hdd: host protected area => 1
* Jan 13 14:13:01 [2] hdd: 80418240 sectors (41174 MB) w/380KiB Cache, CHS=79780/16/63, UDMA(33)
* Jan 13 14:13:01 [4] hdc: attached ide-cdrom driver.
* Jan 13 14:13:01 [2] hdc: ATAPI 40X CD-ROM drive, 128kB Cache
* Jan 13 14:13:01 [2] Uniform CD-ROM driver Revision: 3.12
* Jan 13 14:13:01 [2] Partition check:
* Jan 13 14:13:01 [2] /dev/ide/host0/bus0/target0/lun0: p1
* Jan 13 14:13:01 [2] /dev/ide/host0/bus0/target1/lun0: p1
* Jan 13 14:13:01 [2] /dev/ide/host0/bus1/target1/lun0: p1
* Jan 13 14:13:01 [2] SCSI subsystem driver Revision: 1.00
* Jan 13 14:13:01 [2] Loading Adaptec I2O RAID: Version 2.4 Build 5
* Jan 13 14:13:01 [2] Detecting Adaptec I2O RAID controllers.
* Jan 13 14:13:01 [2] Red Hat/Adaptec aacraid driver (1.1-3 Jul 16 2008 21:50:42)
* Jan 13 14:13:01 [4] scsi: Detection failed (no card)
* Jan 13 14:13:01 [3] megaraid: v2.10.10.1 (Release Date: Thu Jan 27 16:19:44 EDT 2005)
* Jan 13 14:13:01 [4] GDT-HA: Storage RAID Controller Driver. Version: 3.04
* Jan 13 14:13:01 [4] GDT-HA: Found 0 PCI Storage RAID Controllers
* Jan 13 14:13:01 [2] scsi1 : SCSI host adapter emulation for IDE ATAPI devices
* Jan 13 14:13:01 [4] 3ware Storage Controller device driver for Linux v1.02.00.037.
* Jan 13 14:13:01 [4] 3w-xxxx: No cards found.
* Jan 13 14:13:01 [1] libata version 1.20 loaded.
* Jan 13 14:13:01 [2] raw1394: /dev/raw1394 device initialized
* Jan 13 14:13:01 [2] sbp2: $Rev: 1074 $ Ben Collins
* Jan 13 14:13:01 [5] ieee1394: sbp2: Driver forced to serialize I/O (serialize_io = 1)
* Jan 13 14:13:01 [2] usb.c: registered new driver usbdevfs
* Jan 13 14:13:01 [2] usb.c: registered new driver hub
* Jan 13 14:13:01 [2] host/uhci.c: USB Universal Host Controller Interface driver v1.1
* Jan 13 14:13:01 [2] PCI: Found IRQ 11 for device 00:07.2
* Jan 13 14:13:01 [2] PCI: Sharing IRQ 11 with 00:10.0
* Jan 13 14:13:01 [2] host/uhci.c: USB UHCI at I/O 0x1400, IRQ 11
* Jan 13 14:13:01 [2] usb.c: new USB bus registered, assigned bus number 1
* Jan 13 14:13:01 [2] hub.c: USB hub found
* Jan 13 14:13:01 [2] hub.c: 2 ports detected
* Jan 13 14:13:01 [2] usb.c: registered new driver usbkbd
* Jan 13 14:13:01 [2] usbkbd.c: :USB HID Boot Protocol keyboard driver
* Jan 13 14:13:01 [2] Initializing USB Mass Storage driver.
* Jan 13 14:13:01 [2] usb.c: registered new driver usb-storage
* Jan 13 14:13:01 [2] USB Mass Storage support registered.
* Jan 13 14:13:01 [2] NET4: Linux TCP/IP 1.0 for NET4.0
* Jan 13 14:13:01 [2] IP Protocols: ICMP, UDP, TCP
* Jan 13 14:13:01 [2] IP: routing cache hash table of 4096 buckets, 32Kbytes
* Jan 13 14:13:01 [2] TCP: Hash tables configured (established 131072 bind 65536)
* Jan 13 14:13:01 [2] NET4: Unix domain sockets 1.0/SMP for Linux NET4.0.
* Jan 13 14:13:01 [3] RAMDISK: NASLite file system found at block 0
* Jan 13 14:13:01 [4] .<6>Freeing initrd memory: 4006k freed
* Jan 13 14:13:01 [4] VFS: Mounted root (ext2 filesystem).
* Jan 13 14:13:01 [2] Mounted devfs on /dev
* Jan 13 14:13:01 [2] Freeing unused kernel memory: 556k freed
* Jan 13 14:13:01 [2] eth0: link up, 100Mbps, full-duplex, lpa 0xC5E1
* Jan 13 14:13:01 [2] kjournald starting. Commit interval 5 seconds
* Jan 13 14:13:01 [2] EXT3-fs: mounted filesystem with ordered data mode.
* Jan 13 14:13:01 [2] kjournald starting. Commit interval 5 seconds
* Jan 13 14:13:01 [2] EXT3 FS 2.4-0.9.19, 19 August 2002 on ide0(3,65), internal journal
* Jan 13 14:13:01 [2] EXT3-fs: mounted filesystem with ordered data mode.
* Jan 13 14:13:01 [2] kjournald starting. Commit interval 5 seconds
* Jan 13 14:13:01 [2] EXT3 FS 2.4-0.9.19, 19 August 2002 on ide1(22,65), internal journal
* Jan 13 14:13:01 [2] EXT3-fs: mounted filesystem with ordered data mode.
* Jan 13 14:13:01 [2] Adding Swap: 1028088k swap-space (priority -1)
* Jan 13 14:20:07 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:08 [4] VM: killing process mt-daapd
* Jan 13 14:20:09 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:10 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:10 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:10 [4] VM: killing process busybox
* Jan 13 14:20:10 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:10 [4] VM: killing process NL2-0B
* Jan 13 14:20:10 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:13 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
* Jan 13 14:20:14 [4] VM: killing process ushare

John

At first glance I see nothing wrong with the syslog other than that you now have 512M of ram and it still runs out of resources with 1.5G of potential memory. I have to conclude that the motherboard exhibits abnormal behavior due to damage. The only way to potentially disprove that is to run memtest successfully overnight for a number of complete cycles. That has to be the next step. Other than that I'm pretty much out of ideas as to what the cause may be.

floyduk, just try another computer if you can. It will save you a lot of unnecessary headache. Bad hardware in this disposable day and age is just not worth messing with.

I second what Dimension has suggested.

I have had a number of times that NL just didn't run the way it should have thanks to hardware issues. Some motherboard/BIOS/processor/chip sets/NICs just don't work right. I have had the best luck with Intel processors running on Intel chip sets. For NICs, one almost can't go wrong with Intel or 3Com. The word is that the Realtec based cards have drivers that need tweeks of the code to function properly in some cases, not the case with the Intel or 3Com NICs. There are allot of cheap asset recovery servers out there for REALLY CHEAP that will totally rock as a NAS server. Most will be dual processor, have REG ECC memory of a 1GB or more (important for stable long term running), and some even include redundant power supplies and RAID cards with cages for fast drive exchanges. Hardware is cheap now, I just setup a guy a NAS server for $400.00 total cost, it came with two 3.00GHz Xeon processors, 2GB REG ECC, four 250GB SATA drives, and a 4Port 3Ware hardware RAID card. Very fast, very cheap, took less than 10Min to set it up.

Cut your losses,

Mike

Ok I hear what you're saying but one of the advertised uses of this product is that it can be used on old spare hardware. That's what I'm doing and that's why I bought it. If I start spending money on new or even new to me hardware to run this on then it's defeated the purpose of the exercise. If I wanted to buy a NAS server with all the right hardware and software I'd buy a complete packaged solution - not build my own out of what bits I had lying around.

Now as it happens I might be able to move the drives into a different computer. But it won't be such a quiet one and it will take me a load MORE effort - and I think you'll agree that based on the testing I've posted so far I've already invested quite a bit of that.

I'll set another memtest going tonight before I go to bed. To be honest though I'm not buying that as a cause. Why would bad memory make a process take up more RAM than it should? I would expect bad RAM to cause the process to die unexpectedly but not for it to use too much RAM. In this case the OS is being called on to allocate lots of RAM and is simply running out. That looks like a controlled and correct OS behaviour to me - not the kind of random fault that bad RAM causes.

I think it's a lot more likely that there's a bug in DAAP that is causing it to allocate too much RAM when it sees something it doesn't understand. Is it possible that having 2 disks (one a mirror of the other) is confusing DAAP? Or could iTunes copy protected files be a problem? Or video files? Is it possible there's a file in my media folder that is somehow tripping DAAP up?

John

To be honest, I doubt most of us could afford a "complete packaged solution" that would be able to hold a candle to NL running on what is now modest server hardware like I describe above. One thing that allot of people fail to realize is that not all hardware is created equal. There are manufacturers that make rock solid parts and systems that just work, then there are those that no matter how much you tweak and adjust and fiddle they just will not run reliably. Tony and Ralph do a damn fine job of making a distro that runs well on a fairly diverse array of hardware. This is no small feat for just a pair of guys and very small margins.

You may not want to hear this but there is a chance that the hardware you are trying to run this on may not want to play nice and simply will not run the OS. In such a case you may be changing or buying new/used hardware to run it on.

I have been in your shoes a couple of times and the only fix was a change of the motherboard, the NIC, or both.

Mike

Ok first of all - I did run another test over night on the latest RAM. I couldn't figure out how to make memtest run tests on both sticks so I took out one of the 256M sticks and tested with just the other single 256M stick in the system. The memtest ran and compelted with no faults found. So according to that at least the RAM and presumably some of the motherboard are working correctly. All my testing since then has been using this same RAM - a single stick of 256M.

Tony has been kind enough to supply me a new testing ISO M2 v1.01 which I've just burned to a CD and tested a few minutes ago. Sadly I'm still seeing the same problem. Here's my test data:

Testing 256Mb RAM 497M swap. RAM is passed fault-free by memtest.

15:36
DAAP 3372
UPNP 27936

15:37
DAAP 3768
UPNP 31104

15:38
DAAP 3948
UPNP 55072

15:39
DAAP 488544
UPNP 62016

15:40
DAAP 677304
UPNP 2368

15:41
DAAP 675744
UPNP 2496

15:42
DAAP died
UPNP 2496

End of the syslog:

# Jan 15 15:36:49 [2] Adding Swap: 509944k swap-space (priority -1)
# Jan 15 15:41:15 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 15 15:41:15 [4] VM: killing process mt-daapd

Sorry Tony - I wish I could report some improvement but it looks like very much the same story as before.

Mike - I hear ya mate. I do understand that not all hardware is created equal. It should be apparent from all the testing I've done so far that programming, OS kernels, building hardware and so on are not new to me. Part of the reason I used an old DELL machine for this project was because typically DELL (desktop) hardware is pretty well matched and reliable. Most of my other machines are hand built including the very well specified FreeNAS server that sits beside the NASlite server. I was a unix system V OS kernel programmer with ICL for a few years too. And though my knowledge of Linux kernel internals is extremely limited my instincts for debugging should still be pretty good I hope.

Now as for moving to different hardware - yes I could probably do that and yes I probably have spare hardware lying around. But I also have a 15 month old baby and a family business to run. If I'm going to commit the time to building that new server and moving the disks then I want to have some confidence that it's the right move or at least a useful investment of my time. I still don't feel convinced that we've identified hardware as a likely cause. I still feel it's more likely that there's something in my media library that is tripping DAAP up. I'm certainly willing to listen to people's differing opinions on the issue and I'm willing to be persuaded. But please don't assume it's easy or convenient for me to invest the time in moving to new hardware.

As regards your point about the quality of Tony and Ralph's work - I absolutely agree. I wouldn't be spending the time trying to make this work if I wasn't a fan of NASLite and the excellent work they've done so far. I'd also like to mention that there are VERY few companies that would go to the trouble of providing a test ISO like Tony did for me today. Thank you guys - you give excellent support.

John