Report of the bug that prevents sbt from executing correctly in the Bifrost IDE
Overview of the Issue
When using the sbt tool (used to compile/package Scala, and start the Metals server for Scala), an issue appears:
sbt thinks that server is already booting because of this exception:
sbt.internal.ServerAlreadyBootingException: java.io.IOException
After this exception is raised, the user is presented with this option:
Create a new server? y/n (default y)
After pressing Y, sbt continues with its normal execution. However, there are some cases in which the Metals server will execute sbt commands without any user interaction, and they will fail (since the user is not able to press Y when the sbt commands asks if it is necessary to create a new server).
This issue prevents the Metals server from working correctly.
Some solutions were offered but the issue can still be replicated in the Bifrost IDE environment. The #6777 issue
Cause
In the case of the Bifrost IDE, this happens only when the project is created on a folder within /home/BlackDiamond/workspace. This workspace directory is a network directory, mounted on a Azure storage account. When sbt is run, sbt tries to create a socket on a child folder of the project folder (which is on the workspace directory). This fails because of the nature of the directories in workspace (they are network directories).
Patch/Temporary Solution
The temporary solution is to use a custom sbt version (an alpha/snapshot), which creates the socket in another location (which is unique for any project). The only file that was modified in the sbt source code was this (BootServerSocket.java), which looks like this after the change:
/* * sbt * Copyright 2011 - 2018, Lightbend, Inc. * Copyright 2008 - 2010, Mark Harrah * Licensed under Apache License 2.0 (see LICENSE) */packagesbt.internal;importjava.io.IOException;importjava.io.InputStream;importjava.io.OutputStream;importjava.io.UnsupportedEncodingException;importjava.lang.reflect.InvocationTargetException;importjava.lang.reflect.Method;importjava.net.Socket;importjava.net.ServerSocket;importjava.net.SocketException;importjava.net.SocketTimeoutException;importjava.nio.file.Files;importjava.nio.file.Path;importjava.nio.file.Paths;importjava.util.Set;importjava.util.concurrent.ConcurrentHashMap;importjava.util.concurrent.ExecutorService;importjava.util.concurrent.Executors;importjava.util.concurrent.Future;importjava.util.concurrent.LinkedBlockingQueue;importjava.util.concurrent.RejectedExecutionException;importjava.util.concurrent.atomic.AtomicBoolean;importjava.util.concurrent.atomic.AtomicInteger;importnet.openhft.hashing.LongHashFunction;importorg.scalasbt.ipcsocket.UnixDomainServerSocket;importorg.scalasbt.ipcsocket.Win32NamedPipeServerSocket;importorg.scalasbt.ipcsocket.Win32NamedPipeSocket;importorg.scalasbt.ipcsocket.Win32SecurityLevel;importsbt.internal.util.Terminal;importxsbti.AppConfiguration;/** * A BootServerSocket is used for remote clients to connect to sbt for io while sbt is still loading * the build. There are two scenarios in which this functionality is needed: * * <p>1. client a starts an sbt server and then client b tries to connect to the server before the * server has loaded. Presently, client b will try to start a new server even though there is one * booting. This can cause a java process leak because the second server launched by client b is * unable to create a server because there is an existing portfile by the time it starts up. * * <p>2. a remote client initiates a reboot command. Reboot causes sbt to shutdown the server which * makes the client disconnect. Since sbt does not start the server until the project has * successfully loaded, there is no way for the client to see the output of the server. This is * particularly problematic if loading fails because the server will be stuck waiting for input that * will not be forthcoming. * * <p>To address these issues, the BootServerSocket can be used to immediately create a server * socket before sbt even starts loading the build. It works by creating a local socket either in * project/target/SOCK_NAME or a windows named pipe with name SOCK_NAME where SOCK_NAME is computed * as the hash of the project's base directory (for disambiguation in the windows case). If the * server can't create a server socket because there is already one running, it either prompts the * user if they want to start a new server even if there is already one running if there is a * console available or exits with the status code 2 which indicates that there is another sbt * process starting up. * * <p>Once the server socket is created, it listens for new client connections. When a client * connects, the server will forward its input and output to the client via Terminal.setBootStreams * which updates the Terminal.proxyOutputStream to forward all bytes written to the * BootServerSocket's outputStream which in turn writes the output to each of the connected clients. * Input is handed similarly. * * <p>When the server finishes loading, it closes the boot server socket. * * <p>BootServerSocket is implemented in java so that it can be classloaded as quickly as possible. */publicclassBootServerSocketimplementsAutoCloseable {privateServerSocket serverSocket =null;privatefinalAtomicBoolean closed =newAtomicBoolean(false);privatefinalAtomicBoolean running =newAtomicBoolean(false);privatefinalAtomicInteger threadId =newAtomicInteger(1);privatefinalFuture<?> acceptFuture;privatefinalExecutorService service =Executors.newCachedThreadPool( r ->newThread(r,"boot-server-socket-thread-"+threadId.getAndIncrement()));privatefinalSet<ClientSocket> clientSockets =ConcurrentHashMap.newKeySet();privatefinalObject lock =newObject();privatefinalLinkedBlockingQueue<ClientSocket> clientSocketReads =newLinkedBlockingQueue<>();privatefinalPath socketFile;privatefinalAtomicBoolean needInput =newAtomicBoolean(false);privateclassClientSocketimplementsAutoCloseable {finalSocket socket;finalAtomicBoolean alive =newAtomicBoolean(true);finalFuture<?> future;privatefinalLinkedBlockingQueue<Integer> bytes =newLinkedBlockingQueue<Integer>();privatefinalAtomicBoolean closed =newAtomicBoolean(false); @SuppressWarnings("deprecation")ClientSocket(finalSocket socket) {this.socket= socket;clientSockets.add(this);Future<?> f =null;try { f =service.submit( () -> {try {Terminal.console().getLines().foreach( l -> {try {write((l +System.lineSeparator()).getBytes("UTF-8")); } catch (finalIOException e) { }return0; });finalInputStream inputStream =socket.getInputStream();while (alive.get()) {try {synchronized (needInput) {while (!needInput.get() &&alive.get()) needInput.wait(); }if (alive.get()) {socket.getOutputStream().write(5);int b =inputStream.read();if (b !=-1) {bytes.put(b);clientSocketReads.put(ClientSocket.this); } else {alive.set(false); } } } catch (IOException e) {alive.set(false); } } } catch (finalException ex) { } }); } catch (finalRejectedExecutionException e) {alive.set(false); } future = f; }privatevoidwrite(finalint i) {try {if (alive.get()) socket.getOutputStream().write(i); } catch (finalIOException e) {alive.set(false);close(); } }privatevoidwrite(finalbyte[] b) {try {if (alive.get()) socket.getOutputStream().write(b); } catch (finalIOException e) {alive.set(false);close(); } }privatevoidwrite(finalbyte[] b,finalint offset,finalint len) {try {if (alive.get()) socket.getOutputStream().write(b, offset, len); } catch (finalIOException e) {alive.set(false);close(); } }privatevoidflush() {try {socket.getOutputStream().flush(); } catch (finalIOException e) {alive.set(false);close(); } } @SuppressWarnings("EmptyCatchBlock") @Overridepublicvoidclose() {if (closed.compareAndSet(false,true)) {if (alive.get()) {write(2);bytes.forEach(this::write);bytes.clear();write(3);flush(); }alive.set(false);if (future !=null) future.cancel(true);try {socket.getOutputStream().close();socket.getInputStream().close();// Windows is very slow to close the socket for whatever reason// We close the server socket anyway, so this should die then.if (!System.getProperty("os.name","").toLowerCase().startsWith("win")) socket.close(); } catch (finalIOException e) { }clientSockets.remove(this); } } }privatefinalObject writeLock =newObject();publicInputStreaminputStream() {return inputStream; }privatefinalInputStream inputStream =newInputStream() { @Overridepublicintread() {if (clientSockets.isEmpty()) returnTerminal.NO_BOOT_CLIENTS_CONNECTED();try {synchronized (needInput) {needInput.set(true);needInput.notifyAll(); }ClientSocket clientSocket =clientSocketReads.take();returnclientSocket.bytes.take(); } catch (finalInterruptedException e) {return-1; } finally {synchronized (needInput) {needInput.set(false); } } } };privatefinalOutputStream outputStream =newOutputStream() { @Overridepublicvoidwrite(finalint b) {synchronized (lock) {clientSockets.forEach(cs ->cs.write(b)); } } @Overridepublicvoidwrite(finalbyte[] b) {write(b,0,b.length); } @Overridepublicvoidwrite(finalbyte[] b,finalint offset,finalint len) {synchronized (lock) {clientSockets.forEach(cs ->cs.write(b, offset, len)); } } @Overridepublicvoidflush() {synchronized (lock) {clientSockets.forEach(cs ->cs.flush()); } } };publicOutputStreamoutputStream() {return outputStream; }privatefinalRunnable acceptRunnable = () -> {try {serverSocket.setSoTimeout(5000);while (running.get()) {try {ClientSocket clientSocket =newClientSocket(serverSocket.accept()); } catch (finalSocketTimeoutException e) { } catch (finalIOException e) {running.set(false); } } } catch (finalSocketException e) { } };publicBootServerSocket(finalAppConfiguration configuration)throwsServerAlreadyBootingException,IOException {finalPath base =configuration.baseDirectory().toPath().toRealPath();finalPath target =base.resolve("project").resolve("target");if (!isWindows) {if (!Files.isDirectory(target)) Files.createDirectories(target); socketFile =Paths.get(socketLocation(base)); } else { socketFile =null; } serverSocket =newSocket(socketLocation(base));if (serverSocket !=null) {running.set(true); acceptFuture =service.submit(acceptRunnable); } else {closed.set(true); acceptFuture =null; } }publicstaticStringsocketLocation(finalPath base) throwsUnsupportedEncodingException {boolean usingAlternativeSocketLocation =true; // ToDofinalPath alternativeSocketLocation =Paths.get("/home","BlackDiamond",".alt-sock-root"); // ToDofinalPath target =base.resolve("project").resolve("target");if (isWindows) {long hash =LongHashFunction.farmNa().hashBytes(target.toString().getBytes("UTF-8"));return"sbt-load"+ hash; } elseif (usingAlternativeSocketLocation) { // ToDo: CHeckfinalPath originalAbsolutePath =target.toAbsolutePath(); // ToDo: CleanfinalPath locationForSocket =alternativeSocketLocation.resolve(Paths.get(originalAbsolutePath.toString().substring(1)));try {Files.createDirectories(locationForSocket); } catch (IOException ex) {System.out.println(ex.getMessage()); }finalPath pathForSocket =locationForSocket.resolve("sbt-load.sock");returnpathForSocket.toString(); } else {returnbase.relativize(target.resolve("sbt-load.sock")).toString(); } } @SuppressWarnings("EmptyCatchBlock") @Overridepublicvoidclose() {if (closed.compareAndSet(false,true)) {// avoid concurrent modification exceptionclientSockets.forEach(ClientSocket::close);if (acceptFuture !=null) acceptFuture.cancel(true);service.shutdownNow();try {if (serverSocket !=null) serverSocket.close(); } catch (finalIOException e) { }try {if (socketFile !=null) Files.deleteIfExists(socketFile); } catch (finalIOException e) { } } }staticfinalboolean isWindows =System.getProperty("os.name","").toLowerCase().startsWith("win");staticServerSocketnewSocket(finalString sock) throwsServerAlreadyBootingException {ServerSocket socket =null;String name =socketName(sock);boolean jni =requiresJNI()||System.getProperty("sbt.ipcsocket.jni","false").equals("true");try {if (!isWindows) Files.deleteIfExists(Paths.get(sock)); socket = isWindows?newWin32NamedPipeServerSocket(name, jni,Win32SecurityLevel.OWNER_DACL):newUnixDomainServerSocket(name, jni);return socket; } catch (finalIOException e) {thrownewServerAlreadyBootingException(e); } }publicstaticBooleanrequiresJNI() {finalboolean isMac =System.getProperty("os.name").toLowerCase().startsWith("mac");return isMac &&!System.getProperty("os.arch","").equals("x86_64"); }privatestaticStringsocketName(String sock) {return isWindows ?"\\\\.\\pipe\\"+ sock : sock; }}
Only the socketLocation method was modified:
To use this custom sbt version, the Scala project must use the sbt version 1.6.3-SNAPSHOT. This can be configured in the project/build.properties file in the project directory. In order to make sure that this is always the case, the Metals version that is used is the 0.11.3-SNAPSHOT version (this is configured globally by the BlackDiamond extension). This Metals version always sets the sbt version to 1.6.3-SNAPSHOT, whenever a new project is created with Metals.
Final Solution
The final solution is explained here. To summarize the explanation: there will be an sbt configuration that handles whether an alternative socket location must be used and which will it be (both of these configurations are hard-coded in the temporary solution, at the start of the socketLocation method). By default, the normal socket location will be used.
For all practical purposes, the temporary solution and the final solution behave in the exact same way, with only one exception: the temporary solution relies on an SNAPSHOT version of sbt and metals. It is much better to rely on a stable version of both. In order to do so, it is necessary to merge the necessary changes into the sbt repository, so that the issue is fixed on an official version of sbt.