DaemonForums  

Go Back   DaemonForums > Miscellaneous > General software and network

General software and network General OS-independent software and network questions, X11, MTA, routing, etc.

Reply
 
Thread Tools Display Modes
  #1   (View Single Post)  
Old 21st July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default serving a local copy of wikipedia on intranet

Hi !

There are ways of setting wikipedia for offline use .. eg. mediawiki.
What's the best way to do so :

0- For your machine only.
1- For an intranet and having it serve other machines

Thank you for sharing ideas , solutions ..

Last edited by daemonfowl; 23rd July 2012 at 08:02 PM.
Reply With Quote
  #2   (View Single Post)  
Old 21st July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
There are ways of setting wikipedia for offline use .. eg. mediawiki.
Commercial products do exist which can be used to download all Wikipedia content. Zipedia is an example.

I am not aware of any free products offering the same functionality.

Shouldn't you be asking this question on MediaWiki's Website? This would not be information that most in the *BSD community would know.
Reply With Quote
  #3   (View Single Post)  
Old 23rd July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks @ocicat.
A big XML of the latest dump (8g ---> 36g uncompressed)
Links:
http://download.wikimedia.org/enwiki...ticles.xml.bz2
I'm not sure if I can make it useful to my case.
What I need is : an offline English wikipedia serving a small intranet via a web server (static content)
Reply With Quote
  #4   (View Single Post)  
Old 23rd July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
What I need is : an offline English wikipedia serving a small intranet via a web server (static content)
You have not disclosed why you want the ability to access Wikipedia offline, but I can only assume that this is due to:
  • sporadic Internet access through an ISP which has problems consistently being up. If this is true, you may want to look for a better ISP.
  • limited bandwidth. If access is too slow, you still may want to look for a better ISP.
The core problem with going forward with an offline solution is that as soon as the Wikipedia content is downloaded, it will be out-of-date. Wikipedia is constantly being updated.
  • Maybe you are fine with content which has already been changed on the live site. Again, you have not disclosed any requirements.
  • You will need to decide how stale your local copy can become before live content needs to be downloaded again. This means you will be repeating the process manually until you figure out how to automate the necessary steps.
It appears you have a lot of work to do in order to make this work irregardless of what kind of download frequency is required. It also seems like you are taking on a lot of work simply due to problems with a flaky ISP. It seems to me that all of your problems would be solved by getting a different ISP, however, the cost may be prohibitive. If this is the case, you need to factor in the costs of manually updating a local copy & what costs will be incurred to automate the entire process.

Again, you have disclosed any details. Why are you going down this path?
Reply With Quote
  #5   (View Single Post)  
Old 23rd July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thank you!
It's not an ISP issue.It's for learning purposes , sort of catching 2 birds with one stone: setting intranet and having static wikipedia serving all hosts.For the time being I need to learn concepts which wont change anytime soon.
What you posted is genuine but consider my case.I have OpenBSD installed on 4 machines and would like to experience with it (ssh/ftp/apache/ etc .. ) while still benefiting from wikipedia content.Is it a false start ? (I chose intranet as it doesn't need a fully qualified domain)
Reply With Quote
  #6   (View Single Post)  
Old 23rd July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
It's not an ISP issue.It's for learning purposes
As long as you find value from the exercise, it doesn't really matter what anyone else thinks.

However, a large problem you have not resolved is how to integrate all this XML which is downloaded into a local server's environment. In other words, how is this XML to be configured into a local Mediawiki server in order to serve up Wikipedia content? You need to gauge how important this problem is, & set a deadline as to how long you will spend on the problem before stopping. If you can resolve it within a few weeks, it may be a good exercise. If it takes a few months, I question if your time could have been spent better on something else.

I will mention an alternative. You may find it useful, & you might not. All that is important is whether you find value in the idea & effort needed to complete it.

Most Unix administrators & developers keep notebooks where they write down useful commands, or recipes on how to accomplish any particular task. This may include partition sizes used on a particular system, & the history of past attempts (along with reasons why they were changed...). Multi-step configurations which may be done so infrequently that they are not memorized is also fair material to include. Think of it as a cheatsheet or cryptic notes that only you find useful. It isn't meant to be used by anyone else, but it's purpose is to help you stay organized & up-to-date as to what work has been done to what system. At best, it would be updated constantly. Otherwise, it's value diminishes quickly. Yet, if you can keep such a record up-to-date, it can be a very helpful tool serving as a log of work done & comprehension gained.

My suggestion is for you to set up a Mediawiki server in which you can create the pages yourself documenting all of your machines emulating a notebook -- how they are configured, & what you tried that works, & what needs to be done next. A list might be maintained of future needs & their prioritization. Being that wiki's are meant for documenting volatile information which changes a lot, this exercise may help you organize your own thoughts, & help you keep track of what you have done to a number of computers. Organization is a really good thing.

As such, this suggestion is perhaps a little easier than configuring a local version of Wikipedia, but I suspect the effort required to create your own electronic notebook will help you more at this point of where you are on the learning curve -- especially if you keep the information updated.

Again, this is only a suggestion.

Last edited by ocicat; 23rd July 2012 at 10:51 PM. Reason: Clean up grammar.
Reply With Quote
  #7   (View Single Post)  
Old 23rd July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks for the offer ! it seems more beneficial.I've installed mediawiki ..
Code:
 # ln -s /var/www/mediawiki /var/www/htdocs/mediawiki
Then lighttpd-mysql to replace Apache .. http://localhost is working but pointing browser to http://localhost/mediawiki doesn't .. when I add /index.php it says Forbidden .. index.php5 opens a dialog box to save a file .. Do I have to chmod/chown anything related ?
(I have met what's needed according to INSTALL :
Quote:
Required software:
* Web server with PHP 5.2.3 or higher.
* A SQL server, the following types are supported
** MySQL 4.0.14 or higher
** PostgreSQL 8.3 or higher
** SQLite
Reply With Quote
  #8   (View Single Post)  
Old 23rd July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
Then lighttpd-mysql to replace Apache ..
Of course, you can try this from the very beginning, however, I would caution you about being overly ambitious. As an example, if something breaks, how do you debug/fix the problem?

It is a lot easier to find the root cause when you have only made 2-3 changes in comparison to 37. I would recommend getting an initial configuration working with the default Apache 1.3 server first. Once you get that working, then substitute in more exotic choices.

When working through tasks for the first time, do only a few changes at a time, then assess whether it worked to that point. Doing lots of wholesale changes is harder to isolate the fundamental problem.
Quote:
Do I have to chmod/chown anything related ?
Looks like you need to do more research.
Quote:
(I have met what's needed according to INSTALL :
...& you expect that the documentation is going to tell you everything?

Welcome to Unix.

Last edited by ocicat; 23rd July 2012 at 11:30 PM. Reason: Clarity.
Reply With Quote
  #9   (View Single Post)  
Old 23rd July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

As for a local copy of wikipedia :
This zim dump is better than the xml :
http://download.kiwix.org/zim/0.9/wi...ic_01_2012.zim
(I was confused .. thinking zim could import or extract .zim files .. the package is found under /usr/ports/productivity)
Reply With Quote
Old 23rd July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks for guidance & invitation .. it's an honor being among you.
Going back to Apache proves your methodology to be correct ! one difficulty at a time+start with default .. It simply worked :-)
Quote:
Nice yellow flower
MediaWiki 1.18.1
LocalSettings.php not found.
Please set up the wiki first.
I have php though
Code:
 
pkg_info | grep php
php-5.3.10          server-side HTML-embedded scripting language
php-gd-5.3.10       image manipulation extensions for php5
php-mysql-5.3.10    mysql database access extensions for php5
clicking on last ----> http://localhost/mediawiki/mw-config/index.php
When I go setting language a box says :
Quote:
Your session data was lost! Check your php.ini and make sure session.save_path is set to an appropriate directory.

Last edited by daemonfowl; 23rd July 2012 at 11:50 PM. Reason: +php
Reply With Quote
Old 23rd July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
When I go setting language a box says :
Quote:
Your session data was lost! Check your php.ini and make sure session.save_path is set to an appropriate directory.
  • Okay, have you found your php.ini, & how is session.save_path set? Sometimes, errors are straight-forward.
  • Quote:
    Going back to Apache proves your methodology to be correct !
    Maybe now you are beginning to understand why several people here have been harping on using a systematic & cautious approach as opposed to slapping together something that "kinda sorta shoulda work". After all, the person who has to figure out what broke is you. You might as well make it simple on yourself to isolate errors.
Maybe we don't have all the answers either, but a systematic approach to problem-solving can make the situation far more understandable.
Reply With Quote
Old 24th July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

It's the first time I touched a php*.*.ini file .. I set session.save_path to "/var/www/htdocs"
but didn't work .. ok what man or FAQ should I read now ???
Reply With Quote
Old 24th July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
It's the first time I touched a php*.*.ini file ..
...which may be more the reason for researching what should be contained in php.ini.
Quote:
I set session.save_path to "/var/www/htdocs"
It sounds like this was tried blindly.
Quote:
ok what man or FAQ should I read now ???
  • This is your project, & part of embarking into uncharted territory means that research is going to be required at some point. Given that you have over 500 messages posted on this site, you should be starting to get a good idea of what is in the FAQ yourself. So, you can determine yourself where in the FAQ is a discussion of PHP.

    ...but I'm a nice guy, & I will give you a hint. What is the purpose of the FAQ, & should PHP be covered at length in an OpenBSD FAQ?
  • As for manpages, PHP was installed as a package and/or port. Are manpages installed too? How do you determine what is installed in an OpenBSD package?

    If you don't know the answer, where would you go to search?
The point of this exercise is twofold. Thinking is a very good skill to develop, & beginning to find information yourself without the need for spoon-feeding is paramount to moving out of the "newbie" category. In other words, you need to learn this yourself. Now is as good a time as any to begin.
Reply With Quote
Old 24th July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks but I was asking about Php/Mediawiki's respective FAQs .. to save time .. I spend more than 2 hours just trying to 'fix' this tiny annoyance .. I tried ~/tmp ,, didn't work .. then sudo mkdir -m 777 /var/temp .. and edited 2 lines in /etc/php.5.3.ini :

session.save_path = "1;/var/temp"
and
session.save_path = "777;/var/temp"

As I said, I have no background knowledge neither in php nor in apache .. so I'm reading .. let the thread rest for a while :-)
Reply With Quote
Old 24th July 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Quote:
to moving out of the "newbie" category
Thank you for encouragement.Yet sadly true , I need to acquire many basic skills before I bid NewbieLand farewell .. I'm stuck : I can't find a solution nor a key to solution & can't ask more lest you'd call it spoonfeeding .. in the end I've learnt much from spoon-feeding .. I have lots of beginner question still unsolved and would feel bad asking them here but again won't find answers elsewhere .. troubled situation .. :-(
Reply With Quote
Old 24th July 2012
ocicat ocicat is offline
Administrator
 
Join Date: Apr 2008
Posts: 2,879
Thanked 190 Times in 160 Posts
Default

Quote:
Originally Posted by daemonfowl View Post
I'm stuck
I can empathize. Everyone here has been there at one time or another.

However, commiserating doesn't advance any closer to a solution, & simply saying "I'm stuck" is actually very ambiguous. What every engineer has to surmise when in unknown territory is "What do I know, & what do I not know?" This helps frame the problem & indicate possible steps to finding a solution. This is also why we have repeatedly asked you to read the following thread for comprehension:

http://www.daemonforums.org/showthread.php?t=596

Until you can articulate (to yourself) where you are stuck, no advancement is going to be made. Until you can articulate to us where you are stuck, we will not be able to provide recommendations.
Quote:
in the end I've learnt much from spoon-feeding ..
As I have mentioned to you elsewhere, we don't typically do projects for others as it requires hardware & time which may not be available.

So, you have a choice. Either abandon the project, or dig deeper. And part of that exploration requires understanding the problem.
Reply With Quote
Old 5th August 2012
backrow backrow is offline
Real Name: Anthony J. Bentley
Shell Scout
 
Join Date: Jul 2009
Location: Albuquerque, NM
Posts: 116
Thanked 10 Times in 4 Posts
Default

Running a local Mediawiki installation is way overkill if you just want to look at pages.

I just tried Evopedia and it seems to work decently.
__________________
Many thanks to the forum regulars who put time and effort into helping others solve their problems.
Reply With Quote
Old 9th August 2012
daemonfowl daemonfowl is offline
bsdstudent
 
Join Date: Jan 2012
Location: DaemonLand
Posts: 834
Thanked 0 Times in 0 Posts
Default

Thanks backrow !
I initially wanted to have a survival local copy of wikipedia (for 2 reasons at least : bad ISP so many net cuts & no mobile connection to rely on) .. then I followed ocicat's advice :
Quote:
My suggestion is for you to set up a Mediawiki server in which you can create the pages yourself documenting all of your machines emulating a notebook -- how they are configured, & what you tried that works, & what needs to be done next
(but I got stuck in the php.ini thing .. :-) )
So it's not just for locally browsing pages .. for this I just put my mirrored websites to htdocs and use Apache .. yet, mediawiki seems to be a different experience :-) ..
Evopedia ? shame on me , I couldn't even install it .. the qt version .. no idea how to install the src/??? .. as for the py version I'm facing a similar php issue as with mediawiki .. It seems I fail too much & sometimes got busy so I realize I went no further since a long while.
Reply With Quote
Old 10th August 2012
da1 da1 is offline
Port Guard
 
Join Date: Feb 2009
Location: Berlin, DE
Posts: 28
Thanked 1 Time in 1 Post
Default

Google is your friend @daemonfowl .
Reply With Quote
Reply

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
OpenBSD101.com is offline now qmemo OpenBSD General 4 30th July 2011 02:26 PM
OS Popularity on Wikipedia Pages vermaden News 10 31st May 2011 06:49 AM
Updating Wikipedia screenshots rpindy OpenBSD General 15 29th May 2011 09:14 PM
ULE scheduler on Wikipedia Weaseal FreeBSD General 7 27th June 2008 02:43 PM
Installing ports offline ClaptonOrient FreeBSD Ports and Packages 7 13th May 2008 03:35 PM


All times are GMT. The time now is 09:38 PM.


Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Content copyright © 2007-2010, the authors
Daemon image copyright ©1988, Marshall Kirk McKusick