A zoom link has been provided via email to the difx-users/difx-developers lists. The meeting will be recorded and made available shortly after conclusion.

• Date: Thursday February 19, 2021, 1400 UTC
• Title: Real-time correlation with DiFX
• Proposed agenda:
  • Moderator: Walter Brisken
  • Real time correlation @ the VLBA (Walter Brisken)
  • Real time correlation with the EVN (Mark Kettenis)
  • Real time correlation with the LBA (Chris Phillips)
  • Real time correlation discussion
  • Any other short items?
  • Planning date/time/topic for next mini-workshop (10 minutes)
    • Proposal: 1 or 2 weeks later. Select topic from available list or a new idea.

Real-time VLBA correlation

e-VLBI with SFXC @ JIVE

?: What about switch routing timeouts caused by one-way streams? [Greg: from the sounds of it, some ethernet switches will see L2 routing timeouts if a server never talks (other than responding to ARP requests.) If the switch doesn't know where a MAC addr is, it sends the packet out on all ports. Ouch for high-bandwidth streams. This will happen if the switch L2 routing timeout is less than the sending server's ARP cache timeout. The solution is for the receiving server (recorder) to occasionally send some packets. For a discussion of this issue in general see https://www.cisco.com/c/en/us/support/docs/switches/catalyst-6000-series-switches/23563-143.html – our problem isn't directly covered, but it's related to the “asymmetric routing” case. Here's a brief VGOS mention of this problem, on slide 28: https://ivscc.gsfc.nasa.gov/meetings/tow2015/Ruszczyk.Sem2.pdf ]

?: What's the loss for IP fragmentation of large packets vs sending small packets which are not fragmented? A: 10%+ loss for fragmentation (one fragment lost = entire packet lost) vs 0.1% to 0.01% for small packets.

?: What if an antenna fails during the run? A: after 1s there is an EOF from vlitebuf, which causes a restart of that antenna at the next scan boundary.

?: Does vlitebuf fill in lost packets with fake invalid packets? A: No, difx is OK with jumps in time

?: Why not generate fill packets, then an antenna that fails can return before the end of the scan? A: difx does need to see an EOF or a packet at the end time or it will hang and not move to the next job. One alternative is to tell vlitebuf what time the end of scan is, so it could EOF at the correct time.

Phil: Could vlitebuf insert fill pattern rather than an EOF? Walter: EOF is used to let know that scan has ended. But could insert fill pattern up to the end of a file.

Geoff Crew: FUSE file system does support seek, and there is a valid use case for seeking within the circular buffer. So it would be nice to support that. Mark: It's better to not support seek at all than a half-working seek!

There were quite a few more that I missed, please feel free to add

?: mtu? A: Don't fragment, that amplifies packet loss. Many networks are actually jumbo-packet clean (!) and if not, the problem is often close to the sender or receiver. JIVE has been using jumbo everywhere for a long time. Jay Blanchard: the VLBA has some tunneling that limits them to 13xx bytes.

• LBA not really doing eVLBI anymore (after doing quite a bit 10-15 years ago)
• Direct connections from the telescope to the correlator (no intermediary) - wouldn't do that given our time over
• Coped excellently when packet loss was negligible, did not work well at all with moderate to severe packet loss (no robustness)
• Did experiment with hybrid VLBI (recording at the same time as sending and doing real time eVLBI); the lack of robustness at the correlator was an impediment.
• Do use remote recording.

https://nrao-edu.zoom.us/rec/share/ePzsAN7VyOpmfpUQtrVvCSEjJWZhox3pGA-BiwNXB03pJgWq8MDIOim-nRLDgR1e.Gg_LH_7WBf6LKNB6

(Names taken from zoom list: apologies to anyone who I couldn't fully decode):

Adam Deller, Walter Brisken, Anthony Sowinski, Rodrigo Amestica, Chris Phillips, David (?), Eugene Cole (ecole?), Frederic Jaron, Geoff Crew, Greg Lindahl, Harro Verkouter, Javier Gonzales, Jay Blanchard, Jessica King (?), Jim Ogle, Leonid Petrov, Mark Kettenis, Nicole Geiger, Phillip Haftings