Curio Studio

The Home of GreatNews
It is currently Thu Nov 27, 2014 1:33 am

All times are UTC - 5 hours




Post new topic Reply to topic  [ 5 posts ] 
Author Message
PostPosted: Thu Aug 11, 2005 5:58 pm 
Offline

Joined: Tue May 31, 2005 8:01 am
Posts: 115
I've been running GN for three weeks and my newsfeed.db grew to over 100Mb since then. It also became increasingly slower with lots of harddisk I/Os. Today I ran a defrag tool and it told me that newsfeed.db is by far the most fragmented file on my ntfs partition broken into 18773 fragments!

Jack you may want to consider pre-allocating space before performing write I/Os to help prevent fragmentation as the file grows. I know of two methods:

- you could allow the user to pre-define the max size of the database (let's say in my case 200Mb). GN would then create and open a new file, efficiently pre-allocate the largest possible chunks (the fewest fragments) of physical media to hold 200Mb of database contents and thereby reduce fragmentation.

- at least preallocate several blocks at once when Gn starts, and release them when the database is closed if they remain unused (perhaps also a user option to define size of blocks).

Alex


Top
 Profile  
 
 Post subject:
PostPosted: Thu Aug 11, 2005 8:10 pm 
Offline
Site Admin
User avatar

Joined: Fri Feb 18, 2005 12:05 am
Posts: 3470
Allocating disk sectors for a file is something totally controled by operating system. Even GN tries to pre-occupy a big chunk of disk space, OS still can allocate them in discrete disk sectors. This is just something applications cannot control without going for very low level driver programming.

Besides defragging the harddisk, you can also use the "Compress Data After Cleanup" option in Cleanup Wizard. This option will try to reorganize the db and should reduce the fragment if hard disk is defragged.

I'm planning to implement auto-cleanup feature that will keep the number of articles in each channel under control. It looks I have to bump up the priority of this task. :)


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 12, 2005 3:21 am 
Offline

Joined: Tue May 31, 2005 8:01 am
Posts: 115
Jack, I didn't mean you should actively try to change the way the OS allocates harddkisk space for GN. The assumption is that you have a harddisk that isn't much fragmented to begin with. Given that, GN tells the OS to allocate a big chunk of harddisk space, in which case the OS will automatically try to use the largest chunks still available.

For example, Total Commander uses pre-allocation before it copies large files from one partition/disk to the other. Winrar pre-allocates space prior decompressing a file. Bittorrent and eDonkey clients, download managers like Flashget, will allocate enough space on your hard disk for the file you want to download, which can sometimes take a couple of seconds depending how large the file is going to be.

Other examples:
http://www.bcarter.com/sap31.htm :
Quote:
One way to deal with fragmentation is to periodically run a defragmentation utility like the Speed Disk component of Norton Utilities. Another way is to put the database all by itself in an empty partition so it can allocate contiguous space as it grows. A third alternative is to use the ALTER DBSPACE statement to preallocate more than enough space for future expansion.

If the ALTER DBSPACE approach is used on a disk that is initially unfragmented then you will avoid future problems. That's because your database file will continue to use contiguous space even if the rest of the disk becomes fragmented.

The ALTER DBSPACE approach also avoids another performance bottleneck where SQL Anywhere must take time to allocate and initialize more disk space every time it runs out.


http://www.sqlservercentral.com/columni ... ersion.asp:
Quote:
Avoiding fragmentation of database files
To keep fragmentation in the database files at a minimum without defragging them you should avoid using the autogrow setting in SQL Server for them. Preallocate enough size for them from the start and when growth still is needed try to increase the size in large chunks at once. Also make sure you keep the database files on separate disks away from other files to make sure logical contigous space is available for them. Separating the data files from the log file is helpful and also important for performance in other ways, since the log file is written to sequentially.


http://www.ianywhere.com/developer/prod ... 000332.htm:
Quote:
Rapidly-changing database files can lead to excessive file fragmentation on the disk, resulting in potential performance problems. Unless you are working with a database with a high rate of change, you do not need to worry about explicitly allocating space for database files. If you are working with a database with a high rate of change, you may pre-allocate disk space for dbspaces or for transaction logs using either Sybase Central or the ALTER DBSPACE statement.


Thank you for making this a priority. I haven't tried the "Compress data after cleanup" feature yet simply because I didn't want to clean-up yet ;) I don't like to globally delete news items older than x days - waiting for GN to have individual feed preferences for max item age.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 12, 2005 6:07 am 
Offline
Site Admin
User avatar

Joined: Fri Feb 18, 2005 12:05 am
Posts: 3470
Alexander wrote:
...waiting for GN to have individual feed preferences for max item age...


Alex, on the 2nd page of channel properties you can specify individual feed's "Cleanup news that are older than ....".


Top
 Profile  
 
 Post subject:
PostPosted: Fri Aug 12, 2005 12:15 pm 
Offline

Joined: Tue May 31, 2005 8:01 am
Posts: 115
Jack, thanks! I don't know how I could miss that one.

Btw, I don't think the fragmentation problem can be solved by the cleaning-up procedure. While it may remove some of the fragmented chunks, new fragments will occur as soon as GN refreshes the feeds and downloads new items.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC - 5 hours


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group