Saturday, January 29, 2011

How do I install 'repeat' on Ubuntu?

This StackOverflow question mentions a unix command called 'repeat'. It sounds like it does exactly what I want. From reading the question and answers, I think the user is on Mac OSX.

However that command is not installed by default on Ubuntu, and I can't find the package to install to get it. What should I install?

  • From the prompt, I'd guess it's a csh builtin.

    And from reading "man csh", that appears to be the case

      repeat count command
               The specified command, which is subject to  the  same  restric-
               tions  as  the  command  in the one line if statement above, is
               executed count times.  I/O  redirections  occur  exactly  once,
               even if count is 0.
    

    So in order to use it, either type "csh" and issue it from the command line, or write your script so that it uses #!/bin/csh as the interpreter at the top. Here are some csh basics to get you started.

    kmarsh : Better yet- don't get started on an obsolete, incompatible shell. Learn real shell programming and write yourself a repeat alias or function in Bash, a (mostly) Posix standard shell.
    Matt Simmons : Eh. I'm a bash guy, but csh doesn't bother me. I know a *lot* of people that would say the exact same thing as you, except change csh to bash and bash to korn. There's a lot of truth to the fact that korn is more advanced than bash. It's all what you're comfortable with and what gets the job done. csh is going to be around for a long, long time
    Joseph Kern : Or you could change csh to bash and bash to zsh.
    Raphink : While quite a few people have their favorite shell, I find that most sysadmins know bash, while there's few that know csh, zsh and others, and since Ubuntu comes with bash by default (for the users at least, it has dash for root), it's still nicer to play with bash when possible. That's just my opinion though.
  • I can't find this command on Ubuntu. It doesn't seem to exist. I even find it very weird that the post on StackOverflow says it's a builtin command when I can't find it on Ubuntu.

    Edit: Like Matt noted, it is a builtin csh command. The following are tips to do quite the same with bash.

    If what you want is to repeat a command n times, you can do that with a loop though:

    for i in {1..n}; do yourcommand; done
    

    For example, to print 100 times "It works", use:

    for i in {1..100}; do echo "It works"; done
    

    If you want to have a repeat function, you could add something like this to your ~/.bashrc:

    function repeat() { 
        local times="$1"; 
        shift; 
        local cmd="$@"; 
    
        for ((i = 1; i <= $times; i++ )); do 
           eval "$cmd"; 
        done 
     }
    

    Source your ~/.bashrc again with . ~/.bashrc and you can call it:

     $ repeat 2 date
    Mon Dec 21 14:25:50 CET 2009
    Mon Dec 21 14:25:50 CET 2009
    
     $ repeat 3 echo "my name is $USER"
    my name is raphink
    my name is raphink
    my name is raphink
    
    Matt Simmons : It's a "shell builtin", which means it's sort of like "echo" in that although there is a /bin/echo, if you just type "echo", it doesn't get executed. bash (or whatever your shell is) has an "echo" command that it runs instead, which prevents the system from having to launch another process.
    Matt Simmons : Although your way works as well
    Raphink : Yes Matt, I read your comment thanks. However, it's not a builtin in bash. The command doesn't exist when I use bash.
    Dennis Williamson : You can avoid calling the external `seq` by using `for ((i = 1; i <= $times; i++ ))`
    Raphink : Yes, that's probably more efficient Dennis, although I find the `seq` syntax more readable somehow.
    Dennis Williamson : You should note that in your "my name is" example, `$USER` is evaluated before the function is called and because of that, something that changes over time wouldn't be reflected during the repeated runs. In order to fix that, you'd have to do `eval "$cmd"` in your function instead of just `$cmd` and use single quotes around the argument to `repeat` to prevent early evaluation. From there, quoting issues just get hairier.
    Raphink : Nice suggestion Dennis. I'll fix my piece of code with this.
    From Raphink
  • You could use watch, which is a standard command available in any shell. For example:

    watch -n 5 date
    
    From Tobu

How to filter http traffic in Wireshark?

I suspect my server has a huge load of http requests from its clients. I want to measure the volume of http traffic. How can I do it with Wireshark? Or probably there is an alternative solution using another tool?

This is how a single http request/response traffic looks in Wireshark. The ping is generated by WinAPI funciton ::InternetCheckConnection() alt text

Thanks!

  • Ping packets should use an ICMP type of 8 (echo) or 0 (echo reply), so you could use a capture filter of:

    icmp
    

    and a display filter of:

    icmp.type == 8 || icmp.type == 0
    

    For HTTP, you can use a capture filter of:

    tcp port 80
    

    or a display filter of:

    tcp.port == 80
    

    or:

    http
    

    Note that a filter of http is not equivalent to the other two, which will include handshake and termination packets.

    If you want to measure the number of connections rather than the amount of data, you can limit the capture or display filters to one side of the communication. For example, to capture only packets sent to port 80, use:

    dst tcp port 80
    

    Couple that with an http display filter, or use:

    tcp.dstport == 80 && http
    

    For more on capture filters, read "Filtering while capturing" from the Wireshark user guide, the capture filters page on the Wireshark wiki, or pcap-filter (7) man page. For display filters, try the display filters page on the Wireshark wiki. The "Filter Expression" dialog box can help you build display filters.

    par : Sorry, I have forgot to mention the details of the "ping" request. This is Windows way of pinging. It seems icmp has no relation to my case.
    par : See the screenshot of the ping in Wireshark I just have attached
    Simeon Pilgrim : I changed the question from 'ping' to 'http' so you answer will not make sense in context, but I +1 because it's a good ping answer.
    From outis
  • It's not a ping. A ping, as already said by outis, is an ICMP echo request. Your trace displays the establishment and immediate termination of an HTTP connection, and that's what InternetCheckConnection() does. The IP in question, 77.222.43.228, resolves to http://repkasoft.com/, which, I guess, is the URL you pass to InternetCheckConnection().

    You can filter traffic with this IP by using capture or display filter host == 77.222.43.228.

  • Using Wireshark 1.2+ , I would run this batch file:

    :: Script to save a wireshark trace
    :: tshark -D to get interface id
    @echo off
    C:
    cd C:\Temp\NetTracing
    set PATH=%PATH%;C:\Program Files\Wireshark
    echo Tracing host 127.1 or 172.1.1.1 or 10.0.0.1
    
    tshark.exe -i 4 -a duration:900 -S -f "tcp port 80" -w trace.cap
    
    From djangofan

best RAID configuration for postgres

I'm purchasing a server with 8 SAS disks to perform database intensive procedures. Currently the main bottleneck is is large index scans in postgres.

I'm currently deciding between 8x300Gb 10k disks or 8x140Gb 15k disks as it would be more convenient to have 200Gb+ Logical space.

The spec sheet for the RAID controller states: "Integrated Hardware RAID-0, -1, -1E, optional RAID-5, -6, -10, -50, -60"

What would be the best RAID configuration, and what choice in disks would be most suitable?

I'm new to configuring RAID and postgres and appreciate the advice.

  • Go for the 8x146GB disks in a big RAID10 array (4 mirrored pairs striped together). This should provide you the best speed in terms of IO access.

    pstanton : does that mean with 4 mirrored pairs the logical disk space would be 146x2=292Gb?
    womble : No, it would be 146GB*4 since you've got four mirrored pairs of 146GB drives (so 584GB, less HDD manufacturer lie factor, filesystem and LVM overhead, etc).
    From rodjek
  • Integrated Hardware RAID-0, -1, -1E, optional RAID-5, -6, -10, -50, -60

    This sounds a little worrisome to me, it sounds like a low-end RAID controller. You want a good RAID controller that can keep up with 8 fast HDDs (that's actually not a given). If you have a fair amount of writes to your DB, then you really want a Battery Backup Unit, and to enable battery-protected write caching on the RAID controller.

    As for RAID disk layout, there are 2 common schools of thought:

    1. 2 disks in mirror for OS, 2 disks in mirror for DB transaction log, 4 disks in RAID 10 for main DB files.
    2. One big RAID 10 array using all disks, and all OS + log + datastore files on this array (see reasoning here, mirrored by BAARF).

    I would rather not take sides on the RAID volume design, it tends to become a bit of a fact-light discussion. Ideally you should experiment with different storage layouts and benchmark them for your specific workload. My gut feel is that all disks in RAID10 is faster and more robust over multiple workloads.

    One last thing, to make sure that OS partitions and RAID stripe boundaries are aligned (see here, Windows centric, but the principle is general). You can do this when you create the partitions.

    pstanton : thanks. i'm assuming the 'optional' part is an upgraded unit which we'll probably opt for. does that still sound low end?
    Jesper Mortensen : @pstanton: Yes, it still sounds low-end, because it could be a license key upgrade, not a new RAID controller. But there is no way for me to tell, you'll have to talk to your vendor about the controller performance, and perhaps battery backup capabilities.
    Chopper3 : I couldn't agree with you more Jesper, this sounds very worrying to me also.
    pstanton : the upgrade for a RAID 10 controller is an IBM ServeRAID M5015, is that worrying?
    pstanton : ... and is that better/worse than the HP Smart Array P410 ?
    Jesper Mortensen : @pstanton: Why don't you talk with your vendors? The IBM M5015 is a recent model, a midrange LSI logic unit, it should be fine for plain RAID10 which isn't so hard on the RAID controller. See http://www.redbooks.ibm.com/abstracts/tips0738.html
  • You should read the information at BAARF the Battle Against Any RAID Five (Four, ...err..., Free). Therefore, the suggestion to go with RAID 10 is good.

    And for database performance, use more faster disks (even if they're smaller) rather than fewer slower disks (even if they're bigger).

  • Don't forget to align your ext3/4 to your RAID stripe/stride size. (man mkfs.ext3/4 -> stride)

    By the way, is there any postgres setting to make its write match the stripe size ?

    (And google for RAID5 write hole)

    Magnus Hagander : Postgres will always do writes in 8Kb blocks. There is a compile time switch to change it, but usually you don't want to be touching that.
    From BenoƮt

Why does my ntpd not work?

Edit

I've tried all your suggestions, but it seems that ntpd just refuse to synchronize to the server.

[vivs@peter-centos ~]$ /usr/sbin/ntpq -np
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================
 192.168.0.30    .LOCL.           1 u   11   64    3    0.984  232732. 20083.2

Does this jitter "20083.2" indicates the time is manually changed?

I've turned off vmware's time synchronization.

Original Question

Here is the status of ntp

[root@peter-centos gw]# /usr/sbin/ntpq -pn
 remote           refid      st t when poll reach   delay   offset  jitter
=============================================
 192.168.0.30    .LOCL.           1 u  153 1024  377    0.950  1905553 274023.
*127.127.1.0     .LOCL.          10 l    9   64  377    0.000    0.000   0.001

You can see that it only synchronize to '127.127.1.0' which is the local clock.

Is it because of the offset it too large?

But after I manually set the date by date command, it still refuse to synchronize to 192.168.0.30

This is may ntp.conf

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict -6 ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org
#server 1.centos.pool.ntp.org
#server 2.centos.pool.ntp.org
server 192.168.0.30 #blf
#broadcast 192.168.1.255 key 42         # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 key 42             # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 key 42  # manycast client

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
#server 127.127.1.0     # local clock
#fudge  127.127.1.0 stratum 10

# Drift file.  Put this in a directory which the daemon can write to.
# No symbolic links allowed, either, since the daemon updates the file
# by creating a temporary in the same directory and then rename()'ing
# it to the file.
driftfile /var/lib/ntp/drift

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8
lkey 8
h the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8
lkey 8
olkey 8
lkey 8
  • That's a really large jitter value (274023). That indicates that you may have tried to change the time manually while ntpd was running. What you should do is stop ntpd, set the time to the correct time, and then restart ntpd.

  • First off, stop ntpd, and try to set the date using ntpdate {server}:

    /etc/init.d/ntp stop
    /usr/sbin/ntpdate 192.168.0.30
    

    Does this set your time correctly? Or does it time out?

    If it times out, try another NTP server:

    /usr/sbin/ntpdate pool.ntp.org
    

    From the high jitter, I would expect the ntpdate to work - once it has, reboot if possible (just restart ntpd if you can't reboot - though many services will get confused by such a time jump), and check ntpq -p again.

    ablmf : From all the answers. It seems that I should use 'ntpdate' to change date time when ntpd is not running. I tired and I found I could update to 192.168.0.30 by ntpdate. BUT, after I started ntpd again, I saw that ntpd is still not synchronize to 192.168.0.30. I've also removed the local clock from ntp.conf. From ntpq -p, I can see that the jitter is quite large. But I am sure I didn't change the date manual when ntpd is running.
  • Ah -- now it becomes clear:

    My machine is installed in vmware workstation. So, form all the answers, I guess maybe the jitter becomes so large is because that vmware adjust the time. I will see if I am right.

    Don't run ntp in a VM. The host computer doesn't guarantee CPU slices, so the VM's clock isn't accurate. As you see, ntp is trying to keep up with what looks to it like a wildly varying external clock and eventually gives up.

    The general answer to this problem is to not run ntp, to install the VMware tools and lock the VM's clock to the host's clock.

    The specific answer depends on the version of Linux you are running. I have some notes on CentOS (probably generally applicable to other RedHat family distributions) here.

    ablmf : There is also a walk around here : http://wiki.centos.org/TipsAndTricks/VMWare_Server I've tired, it works!
  • VMware has best practices for Linux timekeeping:

    http://kb.vmware.com/selfservice/microsites/search.do?language=en%5FUS&cmd=displayKC&externalId=1006427

    From Tom Kyle

How do I connect a 2008 server to a 2003 server active directory?

Our DC is running Windows Server 2003.

I've just set up Windows Server 2008 and have terminal server running on it. When setting the terminal server permissions, it was able to allow a group name that was read from the domain. In the DC the new terminal server shows up as a computer in the domain.

I can also log in as a user within the domain even though that user doesn't exist locally on the new server.

However, when I go to set sharing permissions on the new machine it doesn't show my domain as a location. Instead it is only looking at location "machinename" and not allowing domain to be seen or added. Is there something I'm missing?

Ok, lots of errors in the event log.

We have this:

The winlogon notification subscriber is taking long time to handle the notification event (Logon).

Followed by this:

The winlogon notification subscriber took 121 second(s) to handle the notification event (Logon).

Followed by:

The processing of Group Policy failed because of lack of network connectivity to a domain controller. This may be a transient condition. A success message would be generated once the machine gets connected to the domain controller and Group Policy has succesfully processed. If you do not see a success message for several hours, then contact your administrator.

I think this might be the same problem I'm having http://serverfault.com/questions/24420/primary-domain-controller-slow

Solved. The issue was that I had changed from DHCP to static and put the wrong DNS server IP in. i.e. firewall instead of DC/DNS server.

  • Make sure you're logged in to the terminal server as a domain account that has administrator rights on the terminal server. Otherwise, the permissions dialogs will default to the local security database.

    Matt : Yes I am logged on as an administrator who has full control to the domain and the terminal server.
    Matt : Having said that, I rebooted the server and now I am seeing hosting.local. But it's not able to search it.
    Matt : No it's gone again, no longer showing hosting.local, just machinename as a location. Very flaky!
  • The issue was with DNS. I had changed the 2008 server from DHCP to static IP and put the wrong DNS server IP address in.

    The DNS server needs to be our domain controller which is also running the DNS server for the Domain in our case. Once I changed that, the terminal server was able to locate the domain controller correctly.

    From Matt

DFS Replication on Lan (with multiple subnets)

target:
want to replicate a folder on 2 machines

situation: we have 2 windows 2003 servers (for this purpose) in a domain and each of these servers has 2 network cards; and each server have 2 ips, one on the LAN and one on the internet like:

20.20.0.100, 192.168.0.100

20.20.0.101, 192.168.0.101

problem: when I use DFS Management tool to create a replication between 2 servers it chooses public ips instead of LAN ips; and our LAN is much much faster. how can I tell the DFS Replication to use LAN ip?

  • I'm not aware of any configuration parameters for DFS-R to control the interfaces that it binds to, or to influence how it selects the partner interface to route traffic to. Doing a quick search, I'm coming up with this dirty hack from the Microsoft Storage Team blog (albeit from 2006) that indicates that you should use a HOSTS file on each replication set member to influence their name resoultion such that you effectively "force" them to use the private IP addresses.

    This is an ugly hack, and I'm typically violently opposed to using HOSTS files. In this case, though, it may well be the only way to accomplish what you're trying to do.

    Rather that doing the HOSTS file hack (which, if you do, you should document so that the next guy who works on it knows why it was done) I have one other idea you might try.

    Try putting a host route for the other host on each of the DFS-R replication set computers. If it works, make the the route persistent. I'm about 80/20 in thinking this won't work versus that it will, but it's worth a shot:

    Member 1: route add 20.20.0.101 mask 255.255.255.255 192.168.0.100

    Member 2: route add 20.20.0.100 mask 255.255.255.255 192.168.0.101

    That might just work to get that traffic flowing over the private network. (If I wasn't under orders from The Wife(tm) to get some house work done this morning I'd give it a try myself and tell you if it works... If she catches me writing on Server Fault this morning it will be bad... >smile<)

  • This is an assumption which will require some investigation on your part, but here's my thought:

    DFS Root and Link targets are identified by host name (although it is possible to create link targets based on ip address instead of host name), those host names are resolved using DNS, if you configure the NIC with the public ip address to not register with DNS (which is how it really should be configured, anyway) then only the internal ip address of the NIC on the LAN will be resolved for the host name of each server, which should force the replication to use the LAN NIC.

    This seems to be pretty close to the solution Evan referenced but without the need to use hosts files or static routes on each host.

    Evan Anderson : It's certainly possible that he wants the public NIC's address registered in DNS for other reasons. If not, though, then this is totally viable. I'd definitely prefer something simple to something ugly like HOSTS files or static routes.
    joeqwerty : I think you misunderstood me. What I mean is that ultimately DFS replication is going to occur between the ip addresses of the hosts that hold the DFS Root and Link targets. If you make sure that only the host names associated to the internal NIC are registered in DNS then when those host names are resolved to their ip address, they'll be resolved to the internal ip address and this may force the replication to use the LAN NIC.
    From joeqwerty
  • I found a way to achieve this!

    I am synching two servers, S1 (Windows Server 2008) and S2 (Windows Server 2008 R2), using DFSR.

    The way I did it was to change the dNSHostName attribute for each Server in Active Directory Users and Computers, to one on the second network card, eg, s1.system.int and s2.system.int.

    It seems DFSR looks at this attribute first before synching, and voila!

    However, I just rebooted and had to set the setting again and don't know yet how to make it stick in AD.

    I also don't know what else uses this dNSHostName attribute, so be warned!

Friday, January 28, 2011

Sharepoint - linked web parts?

I have a page with 2 web parts.

One of them is a list, where the users can add an item (their personal info).

Once they have entered their personal info, it should show up in the 2nd list (with the option to edit)

So in the end, users should be able to see the info of all the users in the 1st list, but only their own in the 2nd list.

Which approach should I use for this? Should both web parts be lists? On the first list I have a "Created By" column that is automatically set to the user's full name when they enter the info, if that helps.

  • Why 2 lists? 1 list, 2 webparts showing that 2 different views of that list. Could even be on different web part pages.

    1st web part shows all items, 2nd web part - Filter the view thusly: Created By = [Me]

    So, in the second list, all you see are items Created by you (or whomever is logged in)