Having fun with Apache Solr opensource search server

Installation is straightforward and hence will not be covered. I use Solr version 5.1.0 and find that the Apache Solr Reference Guide 5.1 does not seems up to date and hence I
have to rely on the help provided by the solr and post binary. This small write-up is for my own record and for anyone who may find it useful.


$ bin/solr create -c test123
Failed to determine the port of a local Solr instance, cannot create test123!

$ bin/solr start -p 8984

Started Solr server on port 8984 (pid=6694). Happy searching!

$ bin/solr create -c test123
WARN  - 2015-05-26 14:52:54.557; org.apache.solr.util.SolrCLI; Request to http://localhost:8984/solr/admin/info/system failed due to: Connection refused, sleeping for 5 seconds before re-trying the request ...

Setup new core instance directory:
/usr/home/fs/solr-5.1.0/server/solr/test123

Creating new core 'test123' using command:
http://localhost:8984/solr/admin/cores?action=CREATE&name=test123&instanceDir=test123

{
  "responseHeader":{
    "status":0,
    "QTime":4023},
  "core":"test123"}

$ bin/solr status

Found 1 Solr nodes:

Solr process 6694 running on port 8984
{
  "solr_home":"/usr/home/fs/solr-5.1.0/server/solr/",
  "version":"5.1.0 1672403 - timpotter - 2015-04-09 10:37:54",
  "startTime":"2015-05-26T02:52:50.118Z",
  "uptime":"0 days, 0 hours, 6 minutes, 29 seconds",
  "memory":"14 MB (%2.9) of 490.7 MB"}

should have start server with default port of 8983, so

$ bin/solr stop

Sending stop command to Solr running on port 8984 ... waiting 5 seconds to allow Jetty process 6694 to stop gracefully.

$ bin/post -c test123 example/exampledocs/*.xml

$ bin/solr start

Started Solr server on port 8983 (pid=7326). Happy searching!

$ bin/post -c test123 example/exampledocs/*.xml

java -classpath /usr/home/fs/solr-5.1.0/dist/solr-core-5.1.0.jar -Dauto=yes -Dc=test123 -Ddata=files org.apache.solr.util.SimplePostTool example/exampledocs/gb18030-example.xml example/exampledocs/hd.xml example/exampledocs/ipod_other.xml example/exampledocs/ipod_video.xml example/exampledocs/manufacturers.xml example/exampledocs/mem.xml example/exampledocs/money.xml example/exampledocs/monitor.xml example/exampledocs/monitor2.xml example/exampledocs/mp500.xml example/exampledocs/sd500.xml example/exampledocs/solr.xml example/exampledocs/utf8-example.xml example/exampledocs/vidcard.xml

SimplePostTool version 5.0.0

Posting files to [base] url http://localhost:8983/solr/test123/update...

Entering auto mode. File endings considered are xml,json,csv,pdf,doc,docx,ppt,pptx,xls,xlsx,odt,odp,ods,ott,otp,ots,rtf,htm,html,txt,log

POSTing file gb18030-example.xml (application/xml) to [base]

POSTing file hd.xml (application/xml) to [base]

POSTing file ipod_other.xml (application/xml) to [base]

POSTing file ipod_video.xml (application/xml) to [base]

POSTing file manufacturers.xml (application/xml) to [base]

POSTing file mem.xml (application/xml) to [base]

POSTing file money.xml (application/xml) to [base]

POSTing file monitor.xml (application/xml) to [base]

POSTing file monitor2.xml (application/xml) to [base]

POSTing file mp500.xml (application/xml) to [base]

POSTing file sd500.xml (application/xml) to [base]

POSTing file solr.xml (application/xml) to [base]

POSTing file utf8-example.xml (application/xml) to [base]

POSTing file vidcard.xml (application/xml) to [base]

14 files indexed.

COMMITting Solr index changes to http://localhost:8983/solr/test123/update...

Time spent: 0:00:01.139

Somehow the link provided in the guide didn't work, but i manage to guess the url to use for query:

go to http://localhost:8983/solr/#/test123/query?q=video to query about video, 


Then i realise i need to remove the # in the url

http://localhost:8983/solr/test123/query?q=video&fl=id,name,price

i got a JSON response:

{
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
      "q":"video",
      "fl":"id,name,price"}},
  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"MA147LL/A",
        "name":["Apple 60 GB iPod with Video Playback Black"],
        "price":[399.0]},
      {
        "id":"EN7800GTX/2DHTV/256M",
        "name":["ASUS Extreme N7800GTX/2DHTV (256 MB)"],
        "price":[479.95]},
      {
        "id":"100-435805",
        "name":["ATI Radeon X1900 XTX 512 MB PCIE Video Card"],
        "price":[649.99]}]
  }}

using http://localhost:8983/solr/test123/select?q=video&fl=id,name,price

i got a XML response

01videoid,name,priceMA147LL/AApple 60 GB iPod with Video Playback Black399.0EN7800GTX/2DHTV/256MASUS Extreme N7800GTX/2DHTV (256 MB)479.95100-435805ATI Radeon X1900 XTX 512 MB PCIE Video Card649.99

To add more documents to be indexed, use the same command as above, i.e.
bin/post -c test123 example/goodies-sg/file.htm


That's all for now...  This is only the beginning, and Solr really has a lot to offer, but so far the steps here should allow me to go further.

Fook Sheng

Vulnerable Web Application for learning how to hack

It used to be only WebGoat and Foundstone's Hacme Bank are the only purpose built vulnerable web application available for people wishing to learn how to hack web application. Now there are more..

Some of those I found lately:

bodgeit
DVWA

Solaris Core dump analysis

A Solaris server running Glassfish keep crashing in my company and even the Oracle consultant has no clue, after he analysed the core dump files.

I have no idea on how to tackle the problem either. But it happen again and again and I decided that it is time for me to learn how to analyse core dump files.

Of course Google is my best friend for a task like this, but some knowledge of reverse engineering helps alot here, without of which might mean getting nothing out even when presented with the correct information.

pflags

Run pflags and look for signals that caused a crash. Here I found a SIGSEGV signal.

pflags core.hostname04.703.26100.java > pflags.txt

and I do a grep and found in the file:

/243:  flags = DETACH
       sigmask = 0xfffffeff,0x0000ffff  cursig = SIGSEGV

The crash occured in thread number 243.

pstack

pstack core.hostname04.703.26100.java > pstack.txt

and look for 243. I got :

-----------------  lwp# 243 / thread# 243  --------------------
ffffffff7dda9840 jni_GetByteArrayRegion (1111471c8, fffffff80b6155c8, 0, 108, fffffff80b6054f8, 111147000) + f8

The function jni_GetByteArrayRegion is causing the crash. It is located in memory address ffffffff7dda9840.

pmap

pmap core.hostname04.703.26100.java > pmap.txt

portion of the pmap.txt:

....

FFFFFFFF7DB12000         16K r----
FFFFFFFF7DC00000       8192K r-x--  /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so
FFFFFFFF7E400000       2048K r-x--  /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so

........

So the culprit is /opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so

I found an easier way to locate the executable that contains the offending function. I will write about it in a later post.

Since the library (/opt/jdk1.6.0_24/jre/lib/sparcv9/server/libjvm.so) is part of the Java Runtime Environment, I have filed a case with Oracle and they are currently investigating. I saved them some work infact :)

Have a nice day :)

fook sheng

Microsoft Windows memory management

Some notes on Microsoft Windows Memory Management (limited to 32 bit OS)

• Maximum memory available is 4G (because Microsoft is dumb enough to ignore segment registers. See http://social.technet.microsoft.com/Forums/en/w7itproperf/thread/9feee84c-44f8-48fa-b49f-9199473e3006)
• 32 bit applications can access up to 4G of memory but in actual fact it can only access 2 G, which is known as the user mode virtual address space (user space)
• User mode address space range from 0000 0000 - 7FFF FFFF
• The other 2G of memory is known as kernel mode address space (system space) and it is reserved for the OS
• Kernel mode address space range from 8000 0000 - FFFF FFFF
• System space contain the global OS code and data structures that are visible to each process

References

http://blogs.technet.com/b/askperf/archive/2007/09/28/memory-management-x86-virtual-address-space.aspx

NFS on MS Windows Server 2003

I am trying out Zen Cloud and a need for NFS as a shared resource arise. My colleague has a Windows server running and i found out that I can run NFS server on the Windows 2003 server.

http://blog.peacon.co.uk/wiki/Creating_an_NFS_Server_using_Windows_Server_2003


In short you need Services for Unix (SFU), which can be downloaded at:

http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=896c9688-601b-44f1-81a4-02878ff11778


The version I had is Windows Services for UNIX Version 3.5.


Happy New Year and cheers :)

/libexec/ld-elf.so.1: Shared object "libidn.so.17" not found, required by "skipf ish"

/libexec/ld-elf.so.1: Shared object "libidn.so.17" not found, required by "skipf
ish"

you got this when u run skipfish...

but

$ ls /usr/local/lib
libidn.a libidn.la libidn.so libidn.so.17 pkgconfig
$

they are there!

by using numerous keywords, and finally using "/usr/lib /usr/local/lib" to google, i found this:


http://ubuntuforums.org/showthread.php?t=289848

Re: /usr/local/lib and /usr/lib

---

iamnafets,

Unless running `ldconfig` is part of the "make install" command for installing the library, your system doesn't know your custom-compiled libraries exist. /usr/local/lib might not be in the library search path either. Do the following as root:

Code:

# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

The above command will append the directory /usr/local/lib to the /etc/ld.so.conf file, creating it if necessary. ldconfig updates the system's cache on libraries installed.

Hope this helps.

the freebsd forum search is hopeless really, but the OS is great.




it still didn't work!

the man pages of ldconfig mention /etc/ld-elf.so.conf



FINALLY


ldconfig /usr/local/lib

it works!



freebsd# ./skipfish
skipfish version 1.31b by
[-] PROGRAM ABORT : Scan target not specified (try -h for help).
Stop location : main(), skipfish.c:394
freebsd#





the man page of ldconfig reads:

SYNOPSIS
ldconfig [-32] [-aout | -elf] [-Rimrsv] [-f hints_file]
[directory | file ...]

DESCRIPTION
The ldconfig utility is used to prepare a set of ``hints'' for use by the
dynamic linker to facilitate quick lookup of shared libraries available
in multiple directories. It scans a set of built-in system directories
and any directories specified on the command line (in the given order)
looking for shared libraries and stores the results in a system file to
forestall the overhead that would otherwise result from the directory
search operations the dynamic linker would have to perform to load the
required shared libraries.

Files named on the command line are expected to contain directories to
scan for shared libraries.....

FreeBSD ports

for example, if i want to install bash shell

become root
cd /usr/ports/shells/bash
make install



OR

make HTTP_PORT=192.168.0.100:8080 install if you are behind a proxy. Of course, in this case, the proxy is 192.168.0.100.