Static File Generation: WSN Support

WSN Support » WSN Links » Static File Generation » Static File Generation

Static File Generation
Can I cronjob it?

Version: 3.16

Static File Generation

Synozeer
Forum Regular

Usergroup: Customer
Joined: Jun 02, 2004
Total Topics: 32
Total Posts: 142

Posted 11/23/05 - 12:30 PM:	#1
As traffic to my site running WSN grows, the server it's on started coming to a crawl. Some testing revealed that the 3 toplists on my index page were the culprist, sometimes bringing my load up to 40+. As soon as I took the toplists down, the load went away. A while back I tried converting one of the toplists into a static file, but whenever the static files were created, it would cause that file to be a 0 byte file. My guess is that because the automatic static file generation routine is going off based on someone visiting the site at the time that routine is supposed to run (instead of going off based purely on a cronjob), the routine is getting run more than once from lots of traffic hitting the site at around that exact moment. I've actually seen the static file being created, then a few moments later a 0 byte file, then created, and then a 0 byte file as the routine is run multiple times. Since I don't think there's a fix for this other than rewriting that portion of the code entirely, I'd like to know how I can set up a cronjob to create the static files instead of using the WSN built in automatic static generation routine. An additional bonus is that I'll be able to run it more than once every 12 hours. Thanks, Adam
GameLinks.com

Paul
Administrator

Usergroup: Administrator
Joined: Dec 21, 2001
Location: Northern California
Total Topics: 57
Total Posts: 6316

Posted 11/23/05 - 03:38 PM:

Do you mean a WSN cron or a server cron? I'll guess server. A wget of

require 'start.php'; 
 if ($settings->staticexporturllist != '') 
 { 
  $lists = explode("\n", decodeit(stripslashes($settings->staticexporturllist), 'full')); 
  $names = explode("\n", $settings->staticexportnamelist); 
  $n = sizeof($lists); 
  for ($x=0; $x<$n; $x++) 
  { 
   $url = trim($lists[$x]); 
   if ($inadmindir) $writeto = '../'; else $writeto = ''; 
   $writeto .= trim($names[$x]);  
   @chmod($writeto, 0666); 
   if (strstr($url, '?')) $url .= '&exporting=1'; else $url .= '?exporting=1'; // make sure it doesn't get into loop 
   $content = geturl($url); 
   $writeit = filewrite($writeto, $content); 
  } 
 } 
require 'end.php';

and then emptying commonfuncs.php's exportstaticpages function to become

function exportstaticpages() 
{ 
 return true; 
}

should do it I suppose, though I've never actually set a cronjob in my life.

the routine is getting run more than once from lots of traffic hitting the site at around that exact moment.

Possible. Hard to verify. Harder to fix since there's no locking mechanism like with mysql.

"Do things you love doing, because then it ain't work. Don't do something you don't really enjoy, because you're never going to work hard enough at it." - Bob Young, founder of Red Hat

Synozeer
Forum Regular

Usergroup: Customer
Joined: Jun 02, 2004
Total Topics: 32
Total Posts: 142

Posted 11/24/05 - 09:28 PM:	#3
Here's the latest on this: I let WSN links try to do the static html by itself (5 jobs - 4 are toplists, 1 is RSS), but it only got through the 1st one before crashing Apache both times. When I got a look at my Apache status while this was happening, it showed one of the links that was pasted in the Static HTML box about 30+ times, so it was indeed trying to run the same routine over and over again. I then created a cronjob on my server to run the above snippet and empited out commonfuncs.php's exportstaticpages routine. This appeared to work correctly... half of the time. The other half of the time, it would create all the files listed in the Static HTML section (which is correct), but then a minute or so later, all sites on the server would stop responding. Server load appeared normal, but when I got a look at active processes and services, NOTHING was showing up - no qmail, no kswapd, or anything else that normally runs. After 15 minutes, all sites were accessible again as if nothing happened. Apache did not go down though. My cron job: 45 5 * * * /usr/bin/php /PATH/TO/PHP SNIPPET >/dev/null 2>&1 My Static HTML jobs (with stuff in CAPS having been renamed for posting): http://www.MYSITE.com/rssfeed.php?type=links&number=20&field=time&ascdesc=descending&thecondition=type='regular'&title=MYSITE http://www.MYSITE.com/index.php?custom=yes&TID=cu...[1]=11&field[1]=rating&ascdesc[1]=descending&type[1]=links http://www.MYSITE.com/index.php?custom=yes&TID=cu...[1]=11&field[1]=hits&ascdesc[1]=descending&type[1]=links http://www.MYSITE.com/index.php?custom=yes&TID=cu...[1]=12&field[1]=time&ascdesc[1]=descending&type[1]=links http://www.MYSITE.com/index.php?custom=yes&TID=cu...[1]=8&field[1]=rand()&ascdesc[1]=descending&type[1]=links (each of these also obviously have an accompanying filename that they are saved as) I didn't use wget - should I be? From the research I did, wget didn't seem to apply here. The most interesting thing is that when I call the snippet from my browser, it works every time without problems. It's only from the cronjob that I have this problem. EDIT: I did a little more research and saw how to properly use WGET. I tried it out, and it appears to be working every time. For some reason, doing the cronjob like I have it causes problems. Here's what I'm now using: 45 * * * * wget -O - http://www.MYSITE.com/PHP SNIPPET Adam Edited by Synozeer on 11/24/05 - 10:27 PM
GameLinks.com

Synozeer
Forum Regular

Usergroup: Customer
Joined: Jun 02, 2004
Total Topics: 32
Total Posts: 142

Posted 11/26/05 - 01:59 PM:	#4
I just wanted to emphasis that this change made a HUGE HUGE HUGE difference. My site (and server) running WSN Links is faster by many multiples. The toplists were taking up a gigantic amount of processing power on my server, and even though I wanted them as Static Files, I couldn't because when the Static Files were being created, it would keep trying to process each hundreds of times, causing my server to crash. Switching to a server cronjob not only fixes that problem, but you can now update your Static HTML pages as often as you want - I'm updating once an hour now. I figured that I'd clarify my results so that anyone who is having speed issues with their site could attempt to fix it. -Adam
GameLinks.com

You don't have permission to post.

Please login or register.