NASLite Network Attached Storage

Hi folks,

I just installed NASLite M2 today over the top of my old NASLite 2 install and for a while it was working great. It still does all the stuff NASLite 2 did just fine.

My NASLite box was always my music drive so all I had to do when I installed M2 was move all the files into the MEDIA folder and turn on DAAP service. Well I've done that and for about 10 mins everything was working great. I see "naslite DAAP" in my list of shared libraries in iTunes and for that first 10 mins I could see and play all the music files in my library (except a few m4a files that didn't seem to have been indexed). But then it stopped working.

So now I can still see "naslite DAAP" in the list of shared libraries but when I click on it I get an error from iTunes saying "The shared library "naslite DAAP" is not responding. (-3260)".

I have iTunes running on both a Mac and a PC. Both versions give the same error. I've rebooted both the server and the clients. It seems safe to assume the problem is at the M2 end.

Any ideas anyone?

John

What are your settings? Did you do anything before you got the error or did it just happen? I had 4 boxes playing mp3 off of m2 last night for over 2 hours with no problems. My wife was getting pissed with me so I had to cut it out.

Do you see any problems in the syslog? After the first test, i dumped over 300G of music in the media folder. It didnt show up a few minutes after i turned it on, but it did later. I guess it take some time to index all the files nefore it can hand them out. I think that was the only confusion i had with it so far.

You need to let M2 index the media, once indexed, they will become available. This process can take longer depending on your indexing type setting. Normal, Fast, Aggressive.

Actually I'm rather assuming the error I'm getting is because something in my music collection is causing the system to choke. The fact that it worked initially suggests that whatever in there is the problem hadn't been been indexed yet.

The problem with this theory is that I would expect a forced re-indexing would prod things back into life at least until the offending item was indexed again. But that isn't what happened. A forced re-index and re-boot did nothing to correct the problem.

I'm doing more testing now. I cleared out the media folder completely. Forced a re-index and rebooted the server then started adding music in a folder at a time. Trust me when I say this is going to take a VERY long time to re-add everything but it's working so far. Of course all that tells me is that the offending item isn't something I've tried adding back yet. Eventually I will and hopefully I'll be able to track down what it is.

No there was nothing in the syslog that gave any clues about where the problem lay.

Thanks for the suggestion about letting M2 index the media but I'm not convinced by that as a cause. If that was the problem then I would have seen it occur during my initial and current testing as I pretty much immediately dropped my entire music collection into the MEDIA folder. Your suggestion, presumably, is that the index is not readable while it's being written to. But right now with very little in my media folder I can drop in a whole bunch more music and carry on using the DAAP share while it's being indexed.

For now this is a very unsatisfactory debugging process that will likely take me days to complete. I would very much appreciate any ideas or suggestions people may have.

John

What is your indexing set at? Normal, Fast or Aggressive?

Each setting will have a different affect on indexing, ie: Fast will just read the file names, aggressive actually scans the entire media file front to back to get time length etc.

I've actually tried a few different settings for the indexing speed. To be honest the PDF manual for M2 doesn't give any satisfactory information on what the settings mean so for a while I had it on aggressive thinking that might make it index more quickly. But at the moment I have it set to the default which seems to be "fast". It would be useful to have more information on what the various settings actually do. Same goes for the background media indexing setting, the consequences of swap area placement and what happens if (as in my case) you have drives being mirrored nightly using RSync - does the DAAP server see the files on both disks?

In my case I have Disk-1 as my main media drive. I also have an identical drive Disk-0 set as a mirror. Rsync runs at 4am daily. Disk-1 is shared read/write, Disk-0 is read only. But of course after tonight both disks will contain the same thing - a MEDIA folder containing all the files. Does DAAP know not to take notice of MEDIA folders on a drive that also contains a MIRROR folder?

John

Hello floyduk,

You raise a valid point about mirrored MEDIA folders. Currently both will be indexed. However, proper behavior should be not to index mirrored media.

I also agree that the manual is lacking and it does need more content covering DAAP and UPNP.

Along with a new kernel and some installer changes, we'll handle both of the above in next release.

Thanks Tony - I think those extra additions to the manual will certainly help.

I'm afraid I'm still unable to get M2 to work, however. I've done a whole bunch more testing and I'm absolutely sure now that the problem is not simply that the media files are still indexing. I started fresh with an empty media folder and added music back 1 folder at a time (I have 1 folder per artist in my case). As I did that the DAAP server was working fine and continued being available in iTunes on both my Mac and PC.

I then tried adding all the rest of the "A"s - this took a while to index. It was probably 15 minutes before iTunes updated the list to show the stuff I'd added. But during that whole time while it was indexing the DAAP service remained available. I then added the "B"s and the same thing happened - 15-20 mins indexing, the DAAP server available all the time. I then tried adding the "C"s, "D"s and "E"s. This time it seemed to work for a short while but then stopped. The DAAP server started giving the "The shared library "naslite DAAP" is not responding. (-3260)" error. So I tried a reboot - no joy. I tried forcing a re-index - no joy. I tried forcing a re-index and then rebooted - no joy. So next I tried removing the "C"s, "D"s and "E"s from the MEDIA folder to see if I could provoke it back to life. It didn't work. I tried another reboot and forced re-index but no joy.

It seems like once whatever it is that is a problem has happened that's it. There's no way to fix it without clearing down the MEDIA folder and starting from square 1. My best guess is that the problem is related to the size of the index - perhaps that in connection with the spec of my server? One thing I have noticed is that I'm seeing something consistant in my syslog. Here's the last couple of lines from my syslog this morning:

# Jan 12 08:33:04 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 08:33:04 [4] VM: killing process mt-daapd

Now you see there that it killed the mt-daapd process. I've also seen the same 0-order allocation failure followed by it killing the klogd process. I don't know if this helps us debug the problem at all but there it is. I'll be happy to provide a complete syslog if anyone wants it.

John

How much ram do you have? How many files roughly? Indexing a ton of media takes lots of ram. Look at the system status page and see what daap is using when indexing. Upnp turns on the swap, so you may want to turn on upnp and see if that helps.

Im running M2 on a Pentium 2 machine with 375Mb of RAM. I run off a CD - M2 is not installed on a hard drive so I assume that means there is no OS swap available. My music collection is 11381 files - about 167Gb

I've noticed now that whenever i reboot my server it says it's starting the DAAP service and the HTTP pages show the DAAP service as running but if I go into a TELNET session and check the services it DOESN'T show DAAP as running in there. So my suspicion is that whatever is causing the problem is causing the DAAP process to die and that's why iTunes can't access it. Why the HTTP pages show it as still running I don't know. Also why it continues to show as an available shared library in iTunes I do not know. Perhaps the SMB service still advertises it without the DAAP back end process being available to serve requests.

I did a whole bunch of googling on this error message in the syslog, "0-order allocation failed" and there are two possibilities I think. First of all we're assuming that the DAAP problems Im having are related to this error. Let's run with that assumption for a moment.

So the error message means that the OS is unable to allocate any more pages of RAM. It's possible that in my case this is exactly the problem. As you see above Im running on 375Mb of RAM with no swap (OS running from cdrom). However, I did also find that MOST of the references to this error I could find seemed to be related to some kind of himem bug in the Linux kernel. So it's possible that might be a cause.

So here's the thing I have to ask - I do have a spare drive in the machine I'm using. Would it likely help me to install M2 on that hard drive? This is quite a bit of effort because I'd have to move a bunch of stuff around so I'd like to be reasonably sure it'll help before I invest the time. Of course it's also worth mentioning that the minimum stated spec for M2 is considerably lower than that of the computer I'm using so it seems reasonable to me to expect it to work just as it is now. If my current spec and configuration can't handle the workload I'm asking of it then I respectfully suggest that the minimum spec should be upped a little. Other customers will likely have similar sized media libraries to my own. Mine isn't even that big compared with some.

Another comment - that tip about turning on uPnP to enable swap.. that's exactly the sort of thing that would go great in the manual as well.

John

A little follow-up information:

I enabled uPnP as per the "enables swap" comment and indeed I now see the system allocating 750Mb of swap according to the HTTP pages. Logging into the TELNET service shows the DAAP and UPNP services are now still both running. HTTP shows DAAP as using about 3Mb of ram and uPnP using about 60Mb. Only just rebooted so these numbers will likely rise. So far no swap is in use. The DAAP share in iTunes is still unavailable and still giving the same error message as before.

UPDATE: after about 5 minutes of running the DAAP and UPNP services have died again. So even with swap space both media indexing service processes died. I'm now losing faith in the idea that installing M2 on a hard disk is going to help. I was assuming that giving the OS some swap space would relieve the memory problems but this test seems to disprove that.

Here's the end of my syslog:

# Jan 12 16:38:33 [2] Adding Swap: 769016k swap-space (priority -1)
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [4] VM: killing process mt-daapd
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [4] VM: killing process ushare
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [4] VM: killing process ushare
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [4] VM: killing process mdnsd
# Jan 12 16:44:14 [3] __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
# Jan 12 16:44:14 [4] VM: killing process mt-daapd

BTW I just realised that on the HTTP status page the light I took as a "service running" indicator is actually a "service enabled" indicator. Subtle but important distinction. So that mystery at least is solved.

John

Hello floyduk,

You don't need to boot from a drive to allow for swap. The swap target location is configurable in the admin menu and defaults to Disk-0. When DAAP bails out, how much RAM do they show as using in the System Status HTML page? Also can you post a syslog as displayed at least 5 minutes after.

Another very important test is to run memtest. Boot the CD and run it for at least 1 full cycle. Memory under load can fail, yet appear fine under normal BIOS scan.

If anyone else is experiencing this problem, please post so we can try and establish some parallels.

Ok Im happy to provide as much information as I can to try and help debug but how do I see the amount of memory being used by DAAP when it crashes? The fastest I can get the HTTP pages to update is once per minute yes? I could tell you the number from the last minute but not the last second.

Regarding the syslog - that extract I just posted was and still is the last thing in the syslog and it's well over an hour later now.

I'll do the memtest.

John

I didn't mean up to the second but rather some time after DAAP dies and the status pages are updated. The memory used should drop down to zero for DAAP. I apologize for the confusion, but I'm unsuccessful in replicating the problem.

I see - well the last time I tested I was watching the memory usage and the last non-zero number I saw for DAAP was in the region of 3Mb. The next number I saw was 0Mb once the process died. The UPNP service number went from about 30Mb to 60Mb and then the next number I saw was 0Mb once it had died. At the time I had the HTTP pages updating every 5 mins. I could re-run the test with the interval set at 1 min if you like.

Just a random thought here. Some of the music in my collection was purchased from the iTunes store and is DRM protected. I had assumed that the media service would simply ignore any files it was unable to play. But has this been tested? Is it possible that these files are causing the DAAP and UPNP services to die? What about video files?

It is particularly interesting to me that both the DAAP and UPNP services died at the same time. It's also interesting that they seemed to run longer with UPNP running (and therefore swap available). That does suggest a memory related problem.

John