Eugene Maker Space Forums
Various Discussions...

Home » Public Forums » Discuss » Re: The Internet Magic at EMS
Re: The Internet Magic at EMS [message #14778] Wed, 23 May 2018 22:00 Go to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
Hi Folks,

I think you're assuming that something was configured to behave this
way. So while the answer to your question is yes the pfsense router's
configuration could be changed, it's not helpful because that's not the
problem.

I played around with the network tonight, and the first thing I did was
swap out the switch for another one. Nothing changed, which was very
very odd....

Finally I started unplugging ports, and lo and behold when I unplugged
the think center on patch-bay port 3 it started working again. This is
super reproducible; plug it in and pull a bunch of dhcp leases. On Linux:

dhclient -v eth0
killall dhclient
repeat over and over

After about a minute it will stop getting leases. Then unplug the think
center and it will start working again. Plug it back in and after a
minute or less it will stop.

So in the future please test the equipment you bring to EMS preferably
at home before you bring it! :-) Unplug everything else and test before
you assume that the network gear is brok
...

Re: The Internet Magic at EMS [message #14779 is a reply to message #14778] Thu, 24 May 2018 06:37 Go to previous messageGo to next message
Michael Bacarella is currently offline  Michael Bacarella
Messages: 64
Registered: June 2017
Member
Hey Clif,

I'm having a bit of trouble parsing your message.

Are you saying someone plugged a thing called a 'think center' into the EMS
network, which broke DHCP for the whole network?

Or are you saying this one particular report about a 'think center' not
working at EMS is explained by the fact that the think center is just
broken and can't work anywhere?

Bill reported like 4 things being broken in this chain, so it's hard to
piece the whole story together.

Is anything still outstanding?


On Thu, May 24, 2018, 01:01 Mr. Clif wrote:

> Hi Folks,
>
> I think you're assuming that something was configured to behave this way.
> So while the answer to your question is yes the pfsense router's
> configuration could be changed, it's not helpful because that's not the
> problem.
>
> I played around with the network tonight, and the first thing I did was
> swap out the switch for another one. Nothing changed, which was very very
> odd....
>
> Finally I started unp
...

Re: The Internet Magic at EMS [message #14780 is a reply to message #14779] Thu, 24 May 2018 06:42 Go to previous messageGo to next message
Ben Hallert is currently offline  Ben Hallert
Messages: 193
Registered: April 2013
Senior Member
I think he's referring to a Lenovo Think Centre PC that was donated, tested, and working for the last few months. Not sure where the shade about testing comes from. ¯\_(ツ)_/¯

That's why we asked if there had been a change to how those ports were routed because David mentioned Clif had done some maintenance upstairs recently in conjunction with the dropout downstairs. If it's a misbehaving computer, then I guess the answer is no. Thanks!

- Ben
________________________________
From: Michael Bacarella
Sent: Thursday, May 24, 2018 6:37:05 AM
To: Clif
Cc: Bruce Newland; Karen Englebeck; board; Ben Hallert; sysadmin; Discuss
Subject: Re: [EMS Board] The Internet Magic at EMS

Hey Clif,

I'm having a bit of trouble parsing your message.

Are you saying someone plugged a thing called a 'think center' into the EMS network, which broke DHCP for the whole network?

Or are you saying this one particular report about a 'think center' not working at EMS is explained by the fact that the think center is just brok
...

Re: The Internet Magic at EMS [message #14781 is a reply to message #14779] Thu, 24 May 2018 06:37 Go to previous messageGo to next message
Michael Bacarella is currently offline  Michael Bacarella
Messages: 64
Registered: June 2017
Member
Sorry, not Bill. I meant Bruce :)

On Thu, May 24, 2018, 09:37 Michael Bacarella
wrote:

> Hey Clif,
>
> I'm having a bit of trouble parsing your message.
>
> Are you saying someone plugged a thing called a 'think center' into the
> EMS network, which broke DHCP for the whole network?
>
> Or are you saying this one particular report about a 'think center' not
> working at EMS is explained by the fact that the think center is just
> broken and can't work anywhere?
>
> Bill reported like 4 things being broken in this chain, so it's hard to
> piece the whole story together.
>
> Is anything still outstanding?
>
>
> On Thu, May 24, 2018, 01:01 Mr. Clif wrote:
>
>> Hi Folks,
>>
>> I think you're assuming that something was configured to behave this way.
>> So while the answer to your question is yes the pfsense router's
>> configuration could be changed, it's not helpful because that's not the
>>
...

Re: The Internet Magic at EMS [message #14782 is a reply to message #14778] Thu, 24 May 2018 08:45 Go to previous messageGo to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
Hi Beata,

yes boots are nice. I think you can get more at Oregon electronics.
Though having the tines break off of the Ethernet connectors is just a
fact of life. I can bring my crimp tool and ends down some time and help
you fix them. Just not this weekend.

    Ciao,
    Clif

On 05/24/2018 12:58 AM, Beata Graff av dalHagen wrote:
>
> Clif,
> Some of the network equipment Is faulty. Specifically the little
> retaining clips on the end of most of the network cables get broken
> off easily. Can you acquire boots and extra crimp fittings? I have a
> bag of boots I can apply to the cause. They are beige. Yellow or blue
> in addition would allow for cable coding if we wanted. And I'm not
> sure how many I have. I need to get some ends to fix some of my own
> cables as well.
>
> Sound like a plan?
>
> On Wed, May 23, 2018, 22:01 Mr. Clif > wrote:
>
> Hi Folks,
>
> I think you're assuming that something was configure
...

Re: The Internet Magic at EMS [message #14783 is a reply to message #14779] Thu, 24 May 2018 08:51 Go to previous messageGo to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
I'm saying someone added something called a 'think center' into the EMS
network which broke way more than DHCP. It basically "jammed" the
switch, not sure by what mechanism.

Not sure if the think center is broken if it was used outside of the EMS
network or not, it is broken in it's current configuration.

Bruce reported all the odd behaviors which made it seem that many things
were broken. Not sure if the wifi problem would have gone away on it's
own, but now I feel that they were all caused by this one machine.

    Clif

On 05/24/2018 06:37 AM, Michael Bacarella wrote:
> Hey Clif,
>
> I'm having a bit of trouble parsing your message.
>
> Are you saying someone plugged a thing called a 'think center' into
> the EMS network, which broke DHCP for the whole network?
>
> Or are you saying this one particular report about a 'think center'
> not working at EMS is explained by the fact that the think center is
> just broken and can't work anywhere?
>
...

Re: The Internet Magic at EMS [message #14784 is a reply to message #14780] Thu, 24 May 2018 09:00 Go to previous messageGo to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
I did come to EMS that day and talk to David. I needed to see something
inside the EMS router, then put it back. I initially plugged one cable
in wrong and David noticed the network went down. I fixed it and checked
that it came back up, and as I wrote before, the network was up when I left.

People have brought stuff to the fair and plugged it in to their part of
the network, which caused the network to go down. This is like stirring
up an ant hill making everyone come running looking for me. Even if I'm
off in a secluded area eating with my family, they will find me, and
drag me back to fix the internet. This is a big hassle for me. Testing
the equipment is important, even if you feel it's already been tested
before.

    Ciao,
    Clif

On 05/24/2018 06:42 AM, Ben Hallert wrote:
> I think he's referring to a Lenovo Think Centre PC that was donated,
> tested, and working for the last few months.  Not sure where the shade
> about testing comes from.  ¯\_(ツ)_/¯
>
>
...

Re: The Internet Magic at EMS [message #14785 is a reply to message #14783] Thu, 24 May 2018 09:24 Go to previous messageGo to next message
Michael Bacarella is currently offline  Michael Bacarella
Messages: 64
Registered: June 2017
Member
On Thu, May 24, 2018 at 11:51 AM, Mr. Clif wrote:

> I'm saying someone added something called a 'think center' into the EMS
> network which broke way more than DHCP. It basically "jammed" the switch,
> not sure by what mechanism.
>
> Jeez, that's bonkers. My no information knee-jerk gut feeling is that it
sounds like a second DHCP server, installed on this 'think center', woke up
and started fighting the EMS DHCP server, confusing every box on the
network. When I get back (early June), I'd love to poke at that think
center, in isolation, with a long stick and see what's going on with it.

This is all to say, sounds maddening. Thanks for tracking that down Clif!

Not sure if the think center is broken if it was used outside of the EMS
> network or not, it is broken in it's current configuration.
>
> Bruce reported all the odd behaviors which made it seem that many things
> were broken. Not sure if the wifi problem would have gone away on it's own,
> but no
...

Re: The Internet Magic at EMS [message #14786 is a reply to message #14785] Thu, 24 May 2018 09:52 Go to previous messageGo to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
Hi Michale,

yes it was maddening, and it took a lot of my time when I don't have any
to spare. I have to drive to the bay area tomorrow to move equipment
around in my rack, then head back to build an antenna tower for some
wifi gear out at the fair.

Yes we've all experienced the rouge DHCP server problem. That's what
happed in my country fair story. However, if you tried to pass SAMBA
traffic on the switch you would have said that SAMBA was failing. If you
tried to pass UDP traffic you would have said that UDP was broken, if
you just tired to ping between two devices that had statically assigned
addresses you would have said ping is broken... See a pattern here? ;-)
When in fact the simple answer is that between many     (*but not all!*
just to make life interesting...)      of the switch ports no traffic
was flowing. Now, you *Could* focus on the DHCP server, or you could
focus on changing the pfsense config, ;-) or a bunch of the other items
that Bruce identified as being broken, but
...

Re: The Internet Magic at EMS [message #14787 is a reply to message #14785] Thu, 24 May 2018 09:58 Go to previous messageGo to next message
Karen Englebeck is currently offline  Karen Englebeck
Messages: 61
Registered: September 2017
Member
The Lenovo Think Centre is a tower-shaped desktop PC with an obnoxious
flange on the top (
https://images-na.ssl-images-amazon.com/images/I/41L5rTv0CbL .jpg). We have
a handful of them at the shop that came in with the donations from
Symantec. In most cases, they should be running Linux Mint Cinnamon (each
having gotten a HD wipe and a brand new install before being deployed. That
said, we did have a couple of computers that refused to boot Linux, and so
they got Windows 7 Pro. I believe they were probably Think Centre's as
well. (I remember having problems with getting the Lenovo's working
properly)

Every machine that has been deployed at the shop, even the Windows
machines, had their HD's reformatted, and the OS installed from
installers... but that doesn't mean someone hasn't installed anything on
them since then.

Of the deployed computers on the network, we have the one at the Vinyl
Cutters (Windows 7), the Mac (Mac OS v?), the one at the 3D Printers
(Linux) and the front desk Kiosk (also Linux). The m
...

Re: The Internet Magic at EMS [message #14788 is a reply to message #14787] Thu, 24 May 2018 10:07 Go to previous messageGo to next message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
Good info,

noting that the think centers were hard to install might be a clue.
Another possibility, though unlikely, is that there is a wiring problem
in a power outlet in the electronics room which causes excessive noise
on that Ethernet cable. The building has three phase power, and you have
to approximate 120 or 240 volts by randomly picking a phase or two.

    Clif

On 05/24/2018 09:58 AM, Karen Englebeck wrote:
> The Lenovo Think Centre is a tower-shaped desktop PC with an obnoxious
> flange on the top
> ( https://images-na.ssl-images-amazon.com/images/I/41L5rTv0CbL .jpg). We
> have a handful of them at the shop that came in with the donations
> from Symantec. In most cases, they should be running Linux Mint
> Cinnamon (each having gotten a HD wipe and a brand new install before
> being deployed. That said, we did have a couple of computers that
> refused to boot Linux, and so they got Windows 7 Pro. I believe they
> were probably Think Centre's as we
...

Re: The Internet Magic at EMS [message #14789 is a reply to message #14786] Thu, 24 May 2018 10:13 Go to previous messageGo to next message
Michael Bacarella is currently offline  Michael Bacarella
Messages: 64
Registered: June 2017
Member
On Thu, May 24, 2018 at 12:52 PM, Mr. Clif wrote:

> Hi Michale,
>
> yes it was maddening, and it took a lot of my time when I don't have any
> to spare. I have to drive to the bay area tomorrow to move equipment around
> in my rack, then head back to build an antenna tower for some wifi gear out
> at the fair.
>

in my country fair story. However, if you tried to pass SAMBA traffic on[/color]
> the switch you would have said that SAMBA was failing. If you tried to pass
> UDP traffic you would have said that UDP was broken, if you just tired to
> machineping between two devices that had statically assigned addresses you
> would have said ping is broken... See a pattern here? ;-) When in fact the
> simple answer is that between many (*but not all!* just to make life
> interesting...) of the switch ports no traffic was flowing. Now, you
> *Could* focus on the DHCP server, or you could focus on changing the
> pfsense config, ;-) or a bunch o
...

Re: The Internet Magic at EMS [message #14790 is a reply to message #14789] Thu, 24 May 2018 10:21 Go to previous message
Clif is currently offline  Clif
Messages: 907
Registered: March 2012
Senior Member
Good questions, my thoughts below...

On 05/24/2018 10:13 AM, Michael Bacarella wrote:

[...]
> Just thinking aloud here while this is still fresh...
>
> Could the rogue box have been spamming the network with (random)
> packets, overwhelming the backplane on the switch?

Yes it's possible, I didn't have time to check for that.

>
> Is focusing on the box the right thing to do here? Is it possible the
> port's misconfigured and any box plugged into it would jam up the switch?
>

It does seem to work fine with everything else plugged in.

> Power issue? N boxes on the network are fine, but N+1 overwhelm the
> switch?

Yes it might be a power issue, as I wrote just a moment ago. Though it
seems unlikely. :-) ...I don't think it's a combination of boxes, I
believe I checked just the think center with the others unplugged, but
it wouldn't hurt to double check that just to be sure. Anyone can do
this check, and it can be done from the electronics room.

     Best
...

Previous Topic: EMS Network down
Next Topic: Nearspace meeting tonight
Goto Forum:
  


Current Time: Wed Aug 15 00:54:34 PDT 2018

Total time taken to generate the page: 0.26215 seconds